CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – III

In the first 2 articles of this series I described a bit of what had to be done on the JS/jQuery client side to trigger the first phase (Phase I) of an Ajax controlled file upload (here: of a Zip container file). See:

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II

In this third article, we take a first glimpse at what has to be done on the server side – in our case in the PHP 5.4 target program of our Ajax request. There are of course very many and different ways to deal with the initial treatment of an uploaded file. What I normally do is to create a “File_Handler” object based on a class which encapsulates and provides all required methods. However, for the sake of a better understandability the code fragments presented below do not always follow a stringent OO line of code development and are sometimes very basic. We only show code elements that are of fundamental importance.

The reader may remember that the method of our Javascript CtrlO for controlling the upload form (see the last article) called a PHP target program named


The attached GET-parameter distinguishes the first phase of the upload from several following phases. One of the initial things our PHP program should do is to open/access a PHP session, to start PHP output buffering and to check whether the $_GET element $_GET[‘file’] exists:

Excerpts from the PHP target program of Phase I – handle_uploaded_init_files.php5:

	$time_0 = microtime(true); 
	// Deal with phase I of the upload 
	if ( isset( $_GET['file'] )) {
		$response = handle_transferred_file(); 
		echo $response;  

(Side remark: Note that the attached Get parameter is nevertheless transferred via the POST mechanism of HTTP !)

$response represents a prepared encoded JSON-Object which is sent back to the JS-Ajax-client at the end of the program. By the “if”-statement we just distinguish the actions of Phase I from later phases where we deal with each file transferred in our large Zip-container file. The main work for our phase I is in our example obviously done inside a function “handle_transferred_file()”.

The use of “session_start()” seems to be quite reasonable. The PHP manual
describes that upload progress information is supplied via $_SESSION. This is just the way the progress tracking of PHP 5.4 works!

However, we shall come back to this point at the bottom of this article and we shall see that things are not that simple. But, we may use the $_SESSION array also for keeping other interesting information. Anyway, using a PHP-session (=opening/accessing a session) will do no harm here.

But why do we need ob_start() ?
In the last article of our series the Ajax answer to the client was requested to have the form of a JSON object. So, our Ajax driven client program expects exactly one information stream in the form of a JSON object. However, if your own PHP code accidentally produces strings by executing some echo, print or print_r statements this rule will be violated. The client will receive the first PHP output as the expected Ajax JSON answer and
will not be able to parse it. This would result in an Ajax client error! Another source of unwanted and not JSON encoded output may come from the PHP engine itself which may create warnings and error messages during code execution. Therefore, it is a good habit to gather all such information in the output buffer and put it eventually as a special string element (as “system_mgs”) of an array which we shall encode as the final JSON object. See below.

A function and an array for producing the JSON answer of upload Phase I

Let us now turn to our function dealing with phase I. In our simple test case it basically does 2 things:

  • It creates a singleton object for performing progress tracking and later some file operations. This object then does the necessary things itself and produces an array which we in our example receive in the variable $response.
  • It enriches the response array with some information and encodes it as a JSON object.

That we have split the required actions into these steps is again more for the sake of illustration purposes.

function handle_transferred_file() {
	// Some local variables 
	$ctrl_msg = '';
	// Includes
	require_once $my_include_path . 'class_file_handler.php5';
	require_once $my_include_path . 'class_file_handler_params.php5';
	// Parameter Singleton with parameters to control the file handling  	
	$F_Params = new FileHandlerParams(); 
	// File handler object (singleton) 	
	$F_Handler = new Basic_Class_File_Handler( $F_Params );
	if (isset($FCheck->msg)) {
		$ctrl_msg .= "\r\nFCheck initialized";
		$ctrl_msg .= "\r\nInitial msg = " . $FCheck->msg;

	// Transfer the file to its directory location and unzip its contents 

	// Required time and response enrichment 		
	$time_f = microtime(true);
	$dtime_f = $time_f - $time_0;
	$F_Handler->ay_ajax_response_1["transfer_time"] = $dtime_f;
	// Response enrichment by system messages form the output buffer 
	$F_Handler->ay_ajax_response_1['sys_msg'] .= ob_get_contents(); 
	// return JSON encoded array 
	$response =json_encode($F_Handler->ay_ajax_response_1);
	return $response; 

This is fairly simple to understand: We load some class definitions – one for the handling of the file transferred into the $_FILE superglobal and one with a bunch of parameters. We then create a singleton object $F_Handler which does most of the required actions of Phase I. Its property “$F_Handler->ay_ajax_response_1” obviously is an array which is enriched by the contents of the PHP output buffer and some time information (just for illustration purposes). This array is eventually encoded as the required JSON answer object.

Typically the element $F_Handler->ay_ajax_response_1[‘sys_msg’] is used on the client side for tests only via the console.log() statement of Javascript.

Possible parameters of the File Handler object $F_Handler

What kind of parameters may be required? E.g.: the path of the directory where we want to save our transferred Zip-file and/or its contents on the server. The expected file ending. The maximum file size allowed. A parameter defining whether we really want to check the progress of the upload. The reader may think about more useful parameters.

In our example such parameters can be gathered and maintained via properties of the class “FileHandlerParams”. The F_Handler object may extract them from the respective Parameter object given as an argument to its constructor, and write them afterward into internal property variables.

Main elements of a Basic_File_Handler class

The “Basic_File_Handler_Class of our example may contain the following methods:

class Basic_Class_File_Handler {
	var $file_key; 		// Key of uploaded file in $_FILES[$key]
	var $file_name = ''; 	// Name of present file (uploaded or part of a pipeline)  
	var $file_mime = ''; 	// Mime-Type of present file 
	var $file_end = ''; 	// Suffix (ending) of present file 
	var $file_expected_end = array("csv", "zip");	// Allowed suffixes of the transferred file 
	var $file_size; 		// Size of present file 
	var $max_file_size = 100000000;	// Maximum allowed size 100MBytes
	var $check_progress = 0; 	// Check progress by means of PHP 5.4  ?
	var $sess_key_progress = ''; 	// The key of $_SESSION where to find upload progress infos 
	// Upload dir 
	var $upl_dir;  			// The given short name of the dir without root path 
	var $target_dir; 		// The full path of the target dir on the PHP server 

	var $ziel_file_name_oe; 	// Target name of file without ending and "."
	var $ziel_file_name; 		// Full target name of file 
	var $ziel_file_pfad; 		// Full target path relative to web domain dir  
	var $ziel_file_pfad_dom;	// Full target path rel. to PHP root on the server 

	var $root;   			// rel. path of present application to PHP application root
	var domain_root;		// rel. path of present application to Web domain root 
	var $upload_success = 0; 
	var $msg = ''; 
	var $sys_msg = ''; 
	var $err = 0; 
	var $err_msg = '';  
	// zip file info 
	var $num_extracted_files = 1; 
	var $file_pipeline = 0;  // will be set to 1 if the zip-file contains more than 1 files 
	// Ajax response array 
	var $ay_ajax_response_1 = array();   // For phase 1 	
// Main methods 
// -----------
	// constructor 
	function __construct($Params)  {

	// function to check properties of the uploaded file and save it in a target directory on the server  
	function check_and_save_uploaded_file() {
	// Method to prepare Ajax (JSON) response array for phase 1 of upload 
	function prepare_ajax_response_phase_1() {
	// Check existence of transferred file in $_FILES
	function check_file_existence_and_props() {
	// Check suffix = file ending 
	function get_and_check_file_suffix() {
	// Check File_Size
	function check_file_size() {
	// Delete all existing files in the upload dir 
	function delete_files_from_upload_dir() {
	// Move the uploaded file or the content files of an uploaded zip file to the target directory 
	function move_file_to_upload_dir() {
	// function to determine the number of extracted files and the name of the next file to handle  
	function determine_num_files_and_next_file_name() {
	// Method to handle a zip archive 
	function handle_zip_archive( $dest_file, $dest_dir) {

// End of class definition  

We shall look at some details of these methods in the articles to come.

The key to session information about the progress of the ongoing upload progress

The constructor of our Basic_File_Handler class gets the task to read in parameters. In addition we use it here and try to retrieve some (initial?) progress information from the $_SESSION array – just for learning purposes. Actually, we shall see that this trial will NOT give us any progress information at all or only a trivial one.

function __construct($Params_ext) {
	// read external parameters 
	$num_args = func_num_args(); 
	if (num_args == 1 ) {
		$Params = func_get_arg(0);
		if ( is_object($Params) && get_class($Params) == "FileHandlerParams" ) {
			$this->file_key  = $Params->file_key; 
			if( is_array($Params->file_types) && count($Params->file_type) > 0 ) {
				$this->file_type = $Params->file_types; 
			$this->upl_dir 	= $Params->upload_dir . "/";
			$this->max_file_size	= $Params->max_file_size; 
			$this->check_progress= $Params->check_progress; 
	// Error treatment 
	else {
		$this->ay_ajax_response_1[sys_msg] .= "\r\nWrong Parameter object!"; 
	// Test the progress information in $_SESSION (Is this reasonable here ???) 
	if ( $this->check_progress == 1 ) {
		// Get the required string end for the key of progress info in the $_SESSION array
		// This information was delivered via $_POST by the Ajax client program 
		$this->sess_key_progress = '';
		$key_POST = ini_get("session.upload_progress.name");
		if (isset( $_POST[$key_POST] ) ) {
			$this->sess_key_progress .= ini_get("session.upload_progress.prefix"). $_POST[$key_POST];
			$this->sys_msg .= "<br>sess_key_progress = " . $this->sess_key_progress;  
			$this->ay_ajax_response_1['sess_key_progress'] = $this->sess_key_progress;  
			$_SESSION['progress_key'] = $this->sess_key_progress;
			// Write a test value into a special Session variable 
			$current = -1;
			if (isset($_SESSION[$this->sess_key_progress]) && !empty($_SESSION[$this->sess_key_progress])) {
				$current 	= $_SESSION[$this->sess_key_progress]["bytes_processed"];
				$total 		= $_SESSION[$this->sess_key_progress]["content_length"];
				$current	= ($current < $total) ? ceil($current / $total * 100) : 100;
			$this->sys_msg .= "<br>Initial progress value of the file transfer was " . $current; 
	// set the TARGET DIR for saving the file
	$this->target_dir 	= $this->root . $this->upl_dir;

$this->check_progress is set by the parameters transferred.

Note that progress tracking and the delivery of related information in the $_SESSION array requires the following setting in the php.ini file (“/etc/php5/apache2/php.ini” on Opensuse)

; Enable upload progress tracking in $_SESSION
; http://php.net/session.upload-progress.enabled
session.upload_progress.enabled = On

In the lower part of the PHP code we see how the key for the element of the $_SESSION array which contains information about the upload progress is composed:

We first need a parameter from the php.ini file which we read by ini_get(“session.upload_progress.prefix”). Then we need another parameter of our php.ini file on the server which gives us a special index of the $_POST array:
$key_POST = ini_get(“session.upload_progress.name”).
This index points to an element of the $_POST array which (hopefully) contains a unique identifier of our presently progressing upload process. We must submit this identifier by our Ajax client method initiating the whole file transfer; see the last article in this series. Eventually, we combine both pieces of information to get the index ($this->sess_key_progress) of an element of the $_SESSION array which will contain progress information about our upload.

In the code above we write this information into our output array as part of an Ajax response.

When and how is the progress information
available in the $_SESSION array?

The next code section then uses our composed key to read a test status value of the upload (at the point of code execution) from the $_SESSION array. Now let us assume that we really used our coding above – what values would we get ? Some integer numbers between 0 and 100 ?

The answer clearly is NO – we would always get “-1”. Why ?

You may say – maybe the connection is too fast and the $_SESSION variables may get erased when the upload has finalized. Well there is some truth in this also:

If we really wanted to test progress tracking in a LAN we may need to reduce the bandwidth of your network connection. Information about how one can achieve this can be found in the article
Geschwindigkeit/Übertragungsrate eines Netzwerkinterfaces unter Linux reduzieren

And yes – depending on the parameter setting in the “php.ini”-file the progress related elements of the $_SESSION array may get eliminated:

; Cleanup the progress information as soon as all POST data has been read
; (i.e. upload completed).
; Default Value: On
; Development Value: On
; Production Value: On
; http://php.net/session.upload-progress.cleanup
session.upload_progress.cleanup = On

But even if you reduced your bandwidth considerably in comparison to your transferred file you would nevertheless only get “-1” ! And even if you set the cleanup parameter to “Off” you would only get a trivial answer: 100.

The real reason for our failure is simply that the code execution of our PHP target program only starts when all data sent via the POST mechanism are completely received – this includes the file data!

This is very logical! The core purpose of the PHP job triggered by our file upload form in the last article is to deal with the uploaded file. It cannot be used for progress tracking by principle! We need a different and independent mechanism – in our case independent polling jobs.

So, when the PHP code listed above is executed on the server the file is already fully uploaded and the $_SESSION elements of $_SESSION which contained the progress information may no longer exist if the cleanup parameter is ON in php.ini settings!

The fact that the PHP job started by the upload form is of no use for progress tracking has two important implications:

First: By what should or could the PHP session for progress tracking be started at all in our Ajax controlled job environment? One assumption could be that the PHP engine does it by itself as soon as it somehow recognizes upload circumstances. However, this is not the case – and uncontrolled automatic session starts would actually introduce security risks into PHP. In fact we have reached an important point :

The PHP session must have been started already before our Ajax controlled file transmission from the client starts!

Otherwise the whole progress tracking process will not work! It uses a PHP session that must have been opened before!

How could we do this? Now, I may remind the kind reader about a note in the last article: Our HTML page with the upload form was created by a PHP program – which itself may use method of the classes of a template engine like Smarty. So, the PHP program that creates our initial web page already could open the required PHP session! An alternative would be that our client starts a precursor Ajax job with the sole purpose of opening a PHP session. Not very efficient, but possible.

And another aspect has become very clear now that would also be valid for independent polling jobs:

If we want to access the progress information in the $_SESSION array about the ongoing upload process we MUST have the $_
POST information about the upload process identifier available as the first information reaching the server – i.e. before the file data themselves start running in! This is the very reason why we had to take care about the order by which data are sent from the client to the server – see the related remarks in the previous article!

Note in addition that we did not destroy the session or unset its cookie at the end of our target program program function for Phase I. The reason is that we may use the session also in later phases.

Enough for today. Please, note also that the only substantial and effective things we have discussed so far on the PHP side were:

  • to access a hopefully already existing (!) PHP session,
  • to initiate the PHP output buffer

However, we have learned something about the key we use for accessing progress information in the $_SESSION array and understood its relation to a $_POST parameter, which must be provided by the client.

We shall come back to the treatment of the fully uploaded file by the methods listed above in a later article.
In the next article

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – IV

we shall instead look at the required sequence of Ajax controlled “polling” jobs started by the client to retrieve information about the upload progress status from the server. Independently of our main PHP job ….

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II

In the first article of this series I summarized which phases and steps we may have to cover during an Ajax driven file upload procedure for a ZIP container file. See:

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I

In this second article, I shall describe in more detail what has to be done during the first step identified for Phase I – the file transfer phase, where a ZIP container file is to be sent over the Internet from the client to the server. We look at the client side and describe some special settings and “tricks” which are required to enable a working and parallel transfer of the ZIP container file and other parameter data to the server via jQuery’s Ajax interface.

Note that the information given below should be regarded as a collection of suggestions. The code snippets are enhanced to make them understandable. Everything can of course be programmed in a different and more efficient way.

We start with some minor preparations to take care of:

Naming conventions for the CSV-files of the Zip container and a related database table

The server must get a chance to distinguish between the required actions for the handling of each of the CSV files in the transferred Zip container. We do this by setting up a small table “fileToTbl” in the LAMP server’s database. We use this table to associate a file name with a target table of the database. A file “alpha.csv” may such be associated with a DB table named “alpha”.

If there are several files referring to a split sequence of data for one and the same target database table the CSV file names may get an integer suffix like “alpha_1.csv, alpha_2.csv, …” – which indicates a loading sequence. The server can extract relevant parts of the file names by the PHP function “explode()” during the treatment of the transferred files.

The table “fileToTbl” may furthermore contain a column “onr” which via an integer defines the standard order by which all possibly received files should be imported into the database. Thus, typical columns would be:

nr, file_name, db_table_name, onr

It is the user’s responsibility to fill only CSV files with the defined names into the ZIP-container to be uploaded. If during ZIP file extraction files are detected which do not fulfill the naming conventions the server should issue a warning in its Ajax response.

Note that although the table “fileToTbl” includes information about all possible files and their names, the number of CSV files send in a specific upload process may be much lower than the maximum possible number. The user may want to update only some input data tables.

Control array for a database import pipeline

The previously described measure enables the server to build up an ordered “pipeline” for loading the received CSV files sequentially into their target tables in Phases II, III, …. When the target PHP program of Phase I server extracts the files from the transferred ZIP-container it can analyze the file names and build up an ordered sequence for the import according to the information found in the table “fileToTbl”.

The information about the loading sequence can be encoded into an array. We call this array “DB Import Control array”, short DBIC-array. It can and will be updated at the end of each Ajax phase and will also exchanged with the client. The (associative) BDIC array should contain information about which files have already been loaded, which files still are to be loaded
and if an error occured. Therefore, elements of the array could be:

dbic[i][‘fname’] = ‘Name_of_the_i-th_file’;
dbic[i][‘fsize’] = ‘Size_of_the_i-th_file’;
dbic[i][‘ftbl’] = ‘Name_of_the_target_DB_table’;
dbic[i][‘ftbl_nr’] = ‘Number_of_the_relevant_record_in_the_fileToTbl_table’;
dbic[i][‘pipe’] = 1; // ( or 0 1: file is part of an import pipeline, 0: only one file,no pipe)
dbic[i][‘loaded’] = 0; // ( or 1 if already loaded)
dbic[i][‘err’] = ‘error_text_or_error_code’;

“i” is an index counting the files according to the loading order defined in “fileToTbl”.

The array will be maintained in PHP’s $_SESSION and shall in addition be part of a JSON object transferred to the client as the Ajax answer of each client/server interaction phase until all files are processed. See again the previous article for the defined phases and steps.

Elements of a simple upload form

Let us now turn the the client and the key preparations there for initializing Phase I. We first build a very simple HTML 4 compatible form to trigger the upload job. We shall extend this web page area later on by progress bars. In the beginning our web page area for the upload shall look as simple as this:


In our project this Web UI area and its contained FORM tag are part of a PHP template (ITX or Smarty TPL) whose placeholder variables are filled by a PHP web page generator program. The TPL handling is assumed to be performed by a specific Template Control Object [TCO] which uses functionality of the chosen template engine. In our examples below we refer to the ITX case where placeholders of the form “{PLACE_HOLDER_NAME}” are to be filled. The template object shall be identified in our PHP code examples by a variable “tpl” of the TCO as $this->tpl.

In the following HTML code of our template we have used speaking IDs. We leave the simple CSS formatting to the reader. Concentrate instead on the FORM tag defined:

<!-- BEGIN UPLOAD -->											
<div id="div_upload_cont">
	<div id="div_upload">
		<div id="upl_float_cont">
			<div id="div_upl_header">
				<p id="upl_header"><span></span></p>
			<div id="csv_file">
				<p><span class="fsnorm">CSV-files: </span><span id="num_open_files" class="bred">0</span></p>
			<div id="imp_file">
				<p><span class="fsnorm">Imported-files: </span><span id="num_extracted_files" class="bgreen">0</span></p>
			<p class="floatstopboth"> </p>
		<form id="form_upload" name="init_file_form"  action="handle_uploaded_init_files.php5" method="POST" enctype="multipart/form-data" >

			<input type="hidden" name="{SESS_UPL_PROGR_FIELD_NAME}" id="hinp_progress_key_name" value="upl">
			<div id="file_cont">
				<input type="file" name="init_file" id="inp_upl_file" >
				<a id="but_submit_upl"  class="basic_but" href="#">Start Upload</a>
				<p class="float_stop"> </p>
			<input type="hidden" name="upl_tbl_num" id="hinp_upl_tbl_num">
			<input type="hidden" name="upl_tbl_name" id="hinp_upl_tbl_name">
			<input type="hidden" name="upl_
tbl_snr" id="hinp_upl_tbl_snr">
			<input type="hidden" name="upl_file_succ" id="hinp_upl_succ" value="0">
			<input type="hidden" name="upl_file_name" id="hinp_upl_file_name" value="0">
			<input type="hidden" name="upl_file_pipe" id="hinp_upl_file_pipe" value="0">
			<input type="hidden" name="rt" id="hinp_upl_run_type">
<!-- END UPLOAD -->

The “init” in some names is unimportant and project specific. In our present context you may assume it indicates Phase I.

The whole area of the enclosing DIV (id=”div_upload_cont”) is controlled on the JS side by an associated Control object “CtrlO_FileUpl” derived from a respective class (see below). The button defined above is used to trigger the file transfer process. This is done by a specific CtrlO method. Therefore, neither a form submit button is used, nor a special “href”-definition is required.

Note the variety of hidden input fields defined inside the FORM tag. Most of these fields will be used in several subsequent Ajax phases until the complete upload process is finalized.

However, during the starting Phase I only the file input field and the first hidden input field are relevant. The first and somewhat special input field can be identified in the code above by a private attribute: lbl=”progress”. (This attribute is otherwise unimportant). The “name” attribute of this input field is determined by a template placeholder which is replaced by the PHP TCO during web page creation.

	<input type="hidden" lbl="progress" name="{SESSION_UPLOAD_PROGRESS_NAME}" id="hinp_progress_name" value="upl">

Note: The position of the first special input was chosen on purpose. We deliver an explanation why the order of the data fields may become important during the POST transfer in a later section of this article.

I want to add a second note here, whose importance we shall understand later in this article series. As we have assumed the web page containing the upload form will be created by a PHP program with the help of a TCO. Now:

The PHP program creating the web page with the template based upload form must initiate a PHP session !

What is the (first) special input field used for?

Our first hidden input field must exist to trigger the initialization of the provision of progress data for the file transfer on the server. The server will look for the name and value of this input variable. The name has to follow rules so that the server recognizes it as special. The value will be used to define a key for accessing progress information in the $_SESSION array.

Actually, the name of the input field has to be identical to the value of the following PHP ini-variable:

; The index name (concatenated with the prefix) in $_SESSION
; containing the upload progress information
; http://php.net/session.upload-progress.name
session.upload_progress.name = “PHP_SESSION_UPLOAD_PROGRESS”

This variable is defined in the php.ini file on the Apache server (normally it is located at “/etc/php5/apache2/php.ini”). It is essential that the server receives data associated with this name in the $_POST or $_GET array; otherwise the PHP engine will not care for the supply of upload progress information.

In a PHP code we can retrieve the value of the required name (from the variable settings in the PHP ini-file) by using the PHP function get_ini(). A PHP code snippet of the page generator TCO for filling our ITX-Block would look like:

 // Treatment of Upload Area 
$upload_block = "UPLOAD"; 
$tpl_hinp_upl_progr_field_name = "SESS_UPL_PROGR_FIELD_NAME"; 
$val_hinp_upl_progr_field_name = ini_get("session.upload_progress.name");
if ( $this->show_upload == 1 ) {
		$this->tpl->setVariable($tpl_hinp_upl_progr_field_name, $val_hinp_upl_progr_field_name);

leading to

<input type="hidden" lbl="progress" name="PHP_SESSION_UPLOAD_PROGRESS" id="hinp_progress_key_name" value="upl">

inside the created web page. The structure of the code snippet would be very similar in case of the SMARTY engine.

Note: The defined value “upl” of the now named input field will later on be used to compose the key which we need to identify the array element containing progress information in the $_SESSION array.

The Javascript CtrlO for the Upload Area

As we have said, we control all event related and all Ajax action – e.g. of course in the initial phase I – by a defined CtrlO on the JS side. Such a singleton object responsible for our upload form may be derived from the following “class” definition:

GOC.CtrlO_FileUpl = new Ctrl_File_Upl('CtrlO_FileUpl');   

function Ctrl_File_Upl(my_name) {
	this.obj_name = "Obj_" + my_name;
	this.GOC = GOC; 
	// Timeout for file transfer process
	this.timeout = 100000; 

	// define selectors of the div and form 
		this.div_upload_cont_sel 	= "#" + "div_upload_cont";
		this.div_upload_sel 		= "#" + "div_upload";
		this.p_header_upload_sel 	= "#" + "upl_header" + " > span";
		this.form_upload_sel 		= "#" + "form_upload";
		this.input_file_sel 		= "#" + "inp_upl_file";
		this.upl_submit_but 		= "#" + "but_submit_upl";
		this.hinp_upl_tbl_num_sel	= "#" + "hinp_upl_tbl_num";			
		this.hinp_upl_tbl_name_sel	= "#" + "hinp_upl_tbl_name";			
		this.hinp_upl_tbl_snr_sel	= "#" + "hinp_upl_tbl_snr";			
		this.hinp_upl_succ_sel 		= "#" + "hinp_upl_succ";			
		this.hinp_upl_run_type_sel 	= "#" + "hinp_upl_run_type";			
		this.hinp_upl_file_name_sel 	= "#" + "hinp_upl_file_name";			
		this.hinp_upl_file_pipe_sel 	= "#" + "hinp_upl_file_pipe";			
		this.num_open_files_sel		= '#' + "num_open_files";
		this.num_extracted_files_sel	= '#' + "num_extracted_files";
	// Determine URL for the Form 
		this.url = $(this.form_upload_sel).attr('action'); 
		console.log("Form_Upload_file - url = " + this.url);  				

	// Register events with jQquery 

// Method to register events   
Ctrl_File_Upl.prototype.register_form_events = function() {
	// this indirectly also calls the secondly defined proxy method below 
		$.proxy(this, 'submit_form') 
	// The real method called 	
		$.proxy(this, 'upl_file') 

We shall look at other methods of such an object later on. Let us first look a bit closer at some of the above definitions.

Side aspects: The GOC object indicated in the first
line is a special object which controls singleton objects in our JS code. I call such an object “Global Object Controller”. Thus we avoid placing the variety of our specific CtrlO objects into the global JS space. Note that such a GOC can also be used for dispatching knowledge about all created singleton CtrlO objects to all objects (e.g. each of the CtrlOs) which may need to know about their existence. Although maybe interesting in itself we do not look at the GOC in detail in this article series.

The variables defined in the beginning of our class definition will be used as jQuery selectors for several objects of our upload container DIV in the CtrlO methods defined below.

The URL of the target PHP program to be addressed by the Ajax request of Phase I is in our example read from the related attribute of the HTML form tag. See the HTML code above for it.

Important: Note the use of jQuery’s $.proxy-mechanism to encapsulate event control in methods of the CtrlO. The definitions given associate a specific event occuring at one of the HTML events with a CtrlO method.

Read more about using $.proxy in the jQuery documentation https://api.jquery.com/jQuery.proxy/. The “trick” here is to define the proper context for the JS “this“-operator used later in the triggered objects methods; the context has to be switched explicitly from the HTML element affected by the user event to the CtrlO object’s method. $.proxy does this for us in a simple, elegant way. You may read more about this trick in another article series of this blog
Fallen beim Statuscheck lang laufender PHP-Jobs mit Ajax – III

We shall look at the central method “upl_file” for starting the upload in a minute.

First obstacle: A method to transfer files and POST data at the same time via jQuery’s Ajax interface

Unfortunately, the transfer of file data from a standard form is not as simple via jQuery’s Ajax interface as soem kind readers may expect. E.g. you run into trouble, if you want to transfer normal input/textarea data from a form together with file (data map) data to a (PHP) server at the same time via the POST mechanism. This is due to the fact that standard settings for the Ajax interface of jQuery may not cover what is required both for file transfer and standard data transfer. Data from standard input elements must be processed to appear in the form of a query string, fitting to the default content-type “application/x-www-form-urlencoded”. However, the corresponding $.ajax settings do not work with file uploads. This is described in the following articles:

What is required to overcome this problem? As described in the named articles a suitable step is to define an internal “FormData” object and attach the information gathered in the relevant input fields of our HTML FORM to this object. The data of the internal “FormData” object are then used in the Ajax controlled transfer with some special parametrization of the Ajax environment (more precise: of the XMLHttpRequest object).

In discussions with some JS developers used to conventional JS coding most find this approach more confusing than helpful. Actually, I personally find it elegant and fully in
line with my general attitude of using internal objects and their methods to control all aspects of user interaction, events and Ajax communication.

The resulting method “upl_file” to start the upload via Ajax looks as follows – note especially the creation of the FormData object :

Ctrl_File_Upl.prototype.submit_form = function (e) {

Ctrl_File_Upl.prototype.upl_file = function(e) { 
	// Prevent Default action 
	// Set cursor to wait 
	$('body').css('cursor', 'wait' ); 

	// Some variables
	var form_data, url;
	// The identification key for the uploaded file in the server's $_Files
	var file_id_key = 'init_file';
	// Reset the values for the numbers of uploaded/open files 
	this.num_extracted_files = 0; 
	this.num_open_files = 0; 
	// Create a FormData object   
	form_data = new FormData();

	// Firstly (!!), add the hidden data to the DataForm object  
	var params = $(this.form_upload_sel).serializeArray();
	$.each(params, function (i, val) {
		form_data.append(val.name, val.value);

	// Secondly, fill the form with the information of the chosen file 
	// for the File API supported in present FF 
	$.each($(this.input_file_sel)[0].files, function(i, file) {
		if (i == 0) {
			form_data.append(file_id_key, file);
		// Multiple file selections in HTML 5 
		$.each($(this.input_file_sel)[0].files, function(key,value) {
			form_data.append(key, value); 

	// "file" will be set and analyzed as a GET parameter to the PHP target url  
	url = this.url + "?file";
	// Time measures 
	this.date_start = new Date(); 
	this.ajax_transfer_start = this.date_start.getTime(); 
	console.log("From Ctrl_File_Upl.success_ajax_file_upl() ::  ajax_start = " + this.ajax_transfer_start);  

	// Setup Ajax 
		// contentType: "application/x-www-form-urlencoded; charset=ISO-8859-1",
		url: url, 
		context:  GOC[this.obj_name],
		timeout: 100000,
		data: form_data, 
		type: 'POST', 
		cache: false, 
		dataType: 'json', 
		contentType: false,
		processData: false, 
		error: this.error_ajax_file_upl,
		success: this.success_ajax_file_upl

Let us discuss some aspects of this method in detail:

  • e.preventDefault is used to prevent that the event triggers a standard reaction of the affected HTML elements. The respective standard event capture and bubbling phases throughout the HTML element hierarchy are interrupted and only the defined CtrlO method code is executed.
  • We define a key name “init_file” for our file to be able later on to identify its precisely in the PHP superglobal array $_FILES. This is not only done for convenience reasons: Although we only upload exactly one file in our present example based on HTML 4.1, we should be prepared to extend our methods to possible multi-file selection options of the modern HTML 5 file upload API.
  • In the beginning of Phase I no files of the ZIP container were processed yet. Therefore, we set the numbers in the respective fields of our form to zero.
  • We create the required internal “FormData”-object.
  • Important:
    We append the special input field defining the key for upload progress information in the $_SESSION array first – i.e.
    before we append any file data.
    Please, do not ignore this point! It took me hours in the beginning to find out that a different order in the data transfer really leads to a complete failure of the whole concept of providing progress data in the $_SESSION superglobal ! Much later and by chance I found a related hint in one of the comments of PHP’s documentation http://php.net/manual/de/session.upload-progress.php

    The point is that the target PHP program of the (Ajax controlled) transfer process receives all data via the POST mechanism – but the PHP 5.4 engine has to recognize already in the very beginning of the transfer that a data upload whose progress shall be followed is initiated. And here an immediate filling of the relevant $_POST-array field is absolutely necessary before the file data appear in the POST buffer. I never had thought about whether there is a ordered sequence of information transfer during the POST process – but there is ! Data for the first fields of a form are transferred first! So, to be consistent always keep the special input field at the top of all other fields providing data in your HTLM FORM tag. In our case the POST data stream is actually derived from the elements of the internal FormData object – but there the same ordering rules are valid. Therefore, we append the data of this input field first to the FormData object.

  • We retrieve the information of the HTML FORM by jQuery’s serialzeArray() functionality. The trick with

    params = $(this.form_upload_sel).serializeArray();

    is that the method $.serializeArray() of the jQuery object does not serialize input data for files or buttons of the HTML FORM tag. So, we only add the values of the hidden input arrays – and among these our special parameter.

  • We now read the information about the file(s) selected in our HTML FORM – we do this already in form of a loop over all possible files of a HTML 5 multiselection field – although we do not really use multiple file selection in our example case. The selector variant in our code is valid for present Firefox browsers which supports a modern HTML5 file API (also in HTML 4.1 code). See e.g.:

    Annotation: It would in our case also have worked without the first [0] as there are no more matching elements for the selector; using the [0] seems however to be good style … For MS IE you need probably version MS IE 10 or 11. I have not tested this.

  • We supply a GET-parameter “file” to our url to explicitly distinguish Phase I from later phases – where we shall define another parameter.
  • Eventually we set up our Ajax environment and trigger the Ajax communication via jQuery’s $.ajax() method.
    Important: Note again that we explicitly set the context for the this-operator of the Ajax interaction environment to our present CtrlO, which itself is an element of the mentioned GOC. Only by using this trick, we can be sure that the “this”-operator in the method used to handle the Ajax response later will
    refer to our present CtrlO. This is really important; otherwise the context would refer to the HTML object triggering the Ajax communication.
  • Note also another the important setting: processData: false
    This prevents jQuery’s Ajax interface from changing data maps (as our file data) into the form of GET variables – thus making it possible to transfer the file data correctly via jQuery’s Ajax interface functions. By setting “contentType: false” we tell the Ajax interface in addition not to care for data types.
  • Last but not least we define methods of our present CtrlO object “GOC.CtrlO_FileUpl” to be responsible for dealing with errors or the Ajax response object in case of a successful communication cycle of our Phase I. We shall look at these methods in a later article.

Note that according to our present setup the target PHP program addressed by the “url” will have to care about input data arriving in the following three superglobals:

$_GET, $_POST and of course $_FILES

Enough for today. We shall see what happens on the server side in the next article of this series to come. At least, we have set up everything such that the server can recognize at the beginning of the data transfer that the progress of the file data transmission shall be tracked.

Please, be a bit patient. The next 3 weeks i am involved in a different project. But the article

CSV file upload with Zip containers, jQuery, Ajax and PHP 5.4 status tracking – III

will be written.

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I

This article series is written in support for French colleagues in a PHP collaboration project and therefore in English. I want to describe some basic elements of an

Ajax controlled file upload process between

  • a browser based User Interface (HTML4/5, Javascript, jQuery)
  • and some PHP/MySQL application programs on a LAMP server.

Our customer’s project depends on a periodic transfer of up to 40 different CSV files with a lot of input data (around 0.5 GByte) to a database server. A requirement of our customer was that the data transfer should be performed with a ZIP file as a container for the individual CSV data files. After the transfer the contents of the individual CSV files should be imported into specific tables of a database.

As our whole interface of the web application is Ajax based, we decided to control all transfers via jQuery’s Ajax API. Meanwhile, there are jQuery Plugins available for this type of task. However, we wanted to fully control all important phases of the file transfer and the data import – both on the client (browser) as well as on the server. This meant that we needed to program all basic steps during an Ajax communication cycle between the browser client and the LAMP server by ourselves. In addition we needed to guarantee some error control.

Personally, I found it a bit astonishing that such a seemingly simple task lead me to some relatively intricate obstacles to overcome. Although most of the necessary ingredients are documented on the Internet, the documentation is sparse and distributed. My objective with this article series is to provide a coherent picture of process design aspects, some coding tricks and also limitations of such a process. This may be useful also for other developers having to solve similar problems. However, if you want to read about a most simple and problem free approach to file upload tasks with Ajax you are probably looking at the wrong article.

Objectives and Assumptions

  • We want to upload several CSV-files (up to 40), whose contents shall be transferred to specific database tables.
  • These files shall be sent to the server in one ZIP file. Reasons for using a ZIP-container file: compression; limitations of the HTML4 file upload API.
  • As the Zip-container may get relatively large we want to see and control the transfer progress over the Internet by some means of PHP 5.4 – as far as possible today.
  • The server shall extract the files from the ZIP and build up a “pipeline” of these files for a subsequent database import of their contents.
  • The data import into database tables shall be done by a sequence of Ajax controlled PHP jobs. Reason: Intermediate information transfer to the client with the option to stop further processing.
  • The server shall decide by some naming conventions what to do with each file.
  • All steps shall be Ajax controlled – a relatively continuous flow of information between client and server has to be established.
  • For the sake of simplicity each Ajax answer of the server at the end of each controlled Ajax transaction cycle shall be encoded in form of a JSON object. (So, if you want to be particularly precise: we use Ajaj instead of Ajax.)

Wording used in this article series

  • JS, jQuery, Ajax:JS below stands for Javascript (on the
    client side). We furthermore use jQuery and its Ajax interface functionality. We expect JSON responses from the server. Although not completely correct we nevertheless use Ajax and Ajaj as synonyms in the articles of this series.
  • Upload: By “upload” we normally mean the whole process. It comprises a “file transfer process” from the Web client PC to the server and subsequent “database import processes”. However, sometimes and for reasons of simplicity we also use the expression “file upload” in a restricted sense – namely for the file transfer to the server, only. It should become clear from the context what we mean.
  • Main PHP program/job:
    The PHP program receiving and working with the transferred Zip-container file and its contents is called “main PHP program” or “main PHP job”. It has to be distinguished from “polling jobs” (see below).
  • Polling jobs: A sequence of additional PHP “polling jobs” may be triggered by the client. This is done in form of a time loop with a short period. A “polling job” on the LAMP server reads some status information of a previously started and still running “Main PHP job” (as e.g. the file transfer job or long lasting database import jobs). The status information of the running main job is fetched from a common data source as the $_SESSION or a database table accessible to both the status writing “main PHP job” and the status reading “polling job”. Each short timed “polling job” fulfills its own complete Ajax transaction cycle. The evaluation of the Ajax response triggers the next polling job if the main PHP job is still running. We come back to the concept of “Ajax driven polling jobs” later on.
  • CtrlOs: Each user interaction area of a web page – e.g. a HTML FORM in a DIV container – shall be completely controlled by a so called JS “Control object” [CtrlO]. A CtrlO encapsulates all reactions of the UI to events in well defined prototype methods. A CtrlO uses jQuery’s proxy mechanism to register events and delegate event handling to defined CtrlO methods. CtrlO methods furthermore control the Ajax communication with the server.
  • Phases: “Phases” describe a full cycle of defined Ajax interaction between client and server. An example of such a full cycle would be:

    HTML Form => Ajax Setup via JS CtrlO method => Submit via JS CtrlO method => POST/FILE data transfer => Server action (PHP) => JSON object as Ajax response => Client analysis of the JSON object via CtrlO method for the Ajax response

  • Client: The client is in our case typically a browser (Firefox) with active JS and jQuery. We do not care about specific requirements of MS IE browsers in this article series; but we assume that at least MS IE browsers > 10 should work.
  • Pipeline: The ordered sequence by which the files of the transferred ZIP container are imported into their related database.

Relevant phases

To get a more detailed overview over what is to be done we distinguish the following main phases and steps (I omit error handling in this overview which may occur at every step):

Phase I – file transfer, progress control and Zip extraction

Step I.1 – Client: Use a HTML form to choose a ZIP file (<input type=”file”>) and use methods of a specifically designed JS
Control object [CtrlO] to control subsequent actions on the client. Add parameter data (hidden input fields) and prepare an Ajax transaction for the file upload (=transfer) process.

Step I.2 – Client: Start the transfer the ZIP file over the Internet to the server. Submit a special parameter in addition to the file to trigger the provision of transfer progress information on the server. Prepare and start the Ajax communication and the data transfer by a CtrlO method.

Step I.3 – Server: Initialize the progress measurement and provide progress data in the $_SESSION array.

Step I.4 – Client/Server: Initiate a sequence of Ajax polling jobs via a JS time loop for reading the progress information on the server. Handle the Ajax response of each polling job in separate defined methods of a special CtrlO. React to error situations and stop the polling job time loop in case of errors or when the file transfer has finalized.

Step I.5 – Server: Extract, expand and save the CSV files from the Zip-container into a special upload folder on the server. This is done by using standard methods of the PHP ZIP class. Define/Suggest a sequence of imports of the data contents of the different files into file specific database tables. This defined sequence may be controlled via an array (“DB Import Control array” = DBIC-array ) which is kept and updated in the $_SESSION array on the server AND which is also sent back via a JSON object to the client.

Step I.6 – Server: Prepare and send an Ajax response in form of a JSON object to the client with affirmation messages about which CSV files have been received, the name of the files and the order in which they shall be processed. Include error messages and system messages if necessary. The JSON object shall contain the “DBIC”-array.

Step I.7 – Client: Analyze the Ajax response. Display success and error information. Display the number and name of files to be processed afterwards. Stop the time loop for polling jobs.

Phase II – Database import of a file in the pipeline

Step II.1 – Client: Prepare and start a new Ajax job with some parameters. The PHP target program of this job shall import the data of one of the already transferred CSV files. Among other things it should be defined, which file shall be processed (= imported into its associated database table) next. This parameter can follow the suggested order of the array which came from the server at the end of Phase I. All parameters can be set up in a separate (hidden) form with hidden input fields. Submit the Ajax job.

Step II.2 – Server: Start the database import on the server with a flexible PHP program. For small and medium sized files (up to approx. below 500000 lines) do it line by line by appropriate special PHP standard methods for handling CSV files. Check the data of each line where reasonable. Gather at least 20 lines in one INSERT statement to accelerate the import process. Write intermediate progress information into a $_SESSION array or a special database table. (This status information may be read by “polling jobs” started on the client.)
For huge files you may extend the import methods later on by using the special MySQL

Step II.3 – Client: Launch a sequence of status information polling jobs via a time loop. Handle the return information of each job in separate methods.

Step II.3 – Server: After a successful import of a defined file remove the file from its upload directory (delete it or move it somewhere else, e.g. in a history directory for uploads). Update you Control Array for the sequence of uploads with the following info: Which of the original files have been loaded? Which had errors? Which are still unprocessed? Prepare a JSON object for an Ajax respond (including the upload Control Array). Send it back to the client.

Step II.4 – Client: Analyze the server’s response. Stop any polling jobs issued after the previous submit. Continue with displaying information of the success of the database import of the handled file. Determine the next file to load. Continue with the elements of Step 5 described above.

Phases III + n – Client/Server – Loop:

Cycle through a sequence of Steps described under II.1 to II.4 for further phases until all files are processed or an error has occurred.

Enough for today. In the next article

CSV file upload with Zip containers, jQuery, Ajax and PHP 5.4 progress tracking – II

we shall cover some major elements of Phase I.