Bash Script to Split Large CSV Files

At times I work with very large CSV files and, unfortunately, not all systems can handle massive file imports. In those cases, I need a quick and easy way to split up one massive CSV file into many smaller CSV files. I’ve written the bash script below to do just that. The approach is based on the one suggested by Mark Setchell on StackOverflow.


#!/bin/bash
# Author: Daniel Ziegler (drziegler.net)

#Accept number of rows to split and filename as command line argument
SPLITNUMBER=$1
FILENAME=$2

#Get filename and file extension (probably .csv or .txt)
BASEFILENAME=${FILENAME%%.*}
FILEEXTENSION=${FILENAME##*.}

#Extract first line of input file as header row
HDR=$(head -1 $FILENAME)

#split file into chunks based on the number of lines input
split -l $SPLITNUMBER $FILENAME num

n=1

#loop through chunks and output to new files in the current directory. Output files will be named “filename-n.ext” where “filename” is the input file name and “.ext” is the extension of that file
for f in num*
do

#include the header row unless we’re looking at the first chunk which already includes the header row
if [ $n -gt 1 ]
then echo $HDR > $BASEFILENAME-${n}.$FILEEXTENSION
fi

cat $f >> $BASEFILENAME-${n}.$FILEEXTENSION

rm $f

((n++))

done

Usage is just ./scriptname.sh XXX filetosplit where “XXX” is the number of lines you want in each smaller file and “filetosplit” is the input CSV file you’re looking to split up.

You can download a Zip of this script here: splitfile.sh

Notes

  • This script assumes that your input file starts with a header row. This header row will be repeated in each output file
  • Should work with any file type, I just use it for CSV files primarily
  • I can’t take any responsibility if this script breaks something! Check the code before you run it!

Restarting a MySQL Server from the Command Line

While it doesn’t come up frequently, every once in a while you’ll need to restart a MySQL server from the command line. On my current webserver we’re running a VPS with WHM. What this means is that accessing the MySQL daemon is not as simple as using “mysqld” from the shell. So here’s how to get it to work, based on information found at this blog post, and my own experience:


#/etc/init.d/mysql start

#/etc/init.d/mysql stop

#/etc/init.d/mysql restart

That will start, stop, and restart, respectively, the MySQL server on our WHM server.

Kiosk/Silent Printing in Google Chrome

As a web developer, the easiest way for me to solve a problem is normally to develop a web app that takes care of the issue. There are certain advantages to this approach: scalability in that multiple users can access the app simultaneously, accessibility in that it can be accessed from anywhere, and hardware independence meaning if the end user’s computer goes down, all I have to do as IT support is replace their computer. At times, however, there are limitations to this “when all you’ve got is a hammer everything looks like a nail” solution. Most often for me these issues are related to functionality which would be extremely simple in a native application being difficult to achieve from a browser. Thankfully, when I encounter these issues I’m also in control of the end user’s browser and computer.

Most recently, I ran into a situation where I needed a web app to print a packing slip for an online order once that order had been confirmed by the picker and packer. The system I’m using gives the packer easy access to only the computer’s number pad as it was much simpler to repurpose hotkeys–”*”, “/”, etc.–from the number pad than try to find space for a mouse on our packing line. Because of this lack of mouse access, however, I needed the system to be able to print without requiring a click on “Print” in Chrome’s print preview dialogue or the system print dialogue. What I discovered thanks to this page on Vend’s website is that Chrome has a little-known kiosk mode with the option for “silent” printing. That is, printing which doesn’t require the user to confirm the printing.

I have Chrome launching on startup for our packers (on a Windows 7 Starter netbook) and all I had to do was right click on the Chrome shortcut in the startup folder, click “Properties” and add the following flags to the end of the “Target:” field:

–kiosk –kiosk-printing

This means that when Chrome launches it’ll launch in full-screen mode (without that annoying “Press F11 to exit full-screen” message everytime your cursor is near the top of the screen) and when you press CTRL-P or call Javascript’s “window.print();” function it will print the page without needing your user to confirm that print action.

As the name of flag suggests, this would also be an extremely useful solution in a case where Chrome were running in a Kiosk computer.

Notes:

A few things to note if you do use this solution to silent printing in Chrome:

  1. “Print Preview” must be enabled in Chrome. Check “about:flags” and make sure.
  2. Slightly annoyingly, Chrome will still show the print preview for about a second before going ahead and printing so it’s not quite silent printing, more like muted printing, but it does not require user confirmation, which was my goal.
  3. This is not a way to auto-print on a users’s computer if you don’t have access to the machine yourself since it requires modifying Chrome’s launch flags. This is a good thing! Imagine visiting an unscrupulous webpage and having it auto-print itself wasting your ink and paper!
  4. Make sure you have the correct printer selected as the default since that’s the one Chrome will print to.
  5. Thanks to Matthew McIntyre for pointing out in a comment here that you can omit the -kiosk flag if you want to use silent printing but don’t want Chrome to open full-screen. Different scenario to mine, but useful to note.