CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – V

In the previous articles of this series about an Ajax controlled file upload with PHP progress tracking

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – IV
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – III
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I

we have studied elements of an Ajax controlled setup to trigger the upload itself and additional independent jobs for progress tracking.

On the Javascript side we first became acquainted with Control objects [CtrlOs] to fully control HTML elements and the Ajax communication of different forms and objects with the PHP server. We then learned that our main PHP job [handle_uploaded_init_files.php5] triggered from the file upload form is of no use for progress tracking.

In the last article we therefore discussed some Javascript methods to start a sequence of independent Ajax controlled polling jobs. Such jobs can retrieve information about the progress of the file upload from the server periodically and in the background. We learned in addition that we may need some adaption of our polling time interval to the measured data transfer rate. We expect to get the rate information from the server together with some standard progress information as e.g. the sum of bytes uploaded at the moment of polling.

The progress information for the file transfer is continuously updated in the $_SESSION array on the LAMP server according to settings in the php.ini configuration file. In this article we now discuss the rather simple PHP job for fetching the information and sending it back to the client (browser).

The PHP polling job to read progress information and send it to the client

The periodically called PHP job, which we named “check_progress.php5”, basically contains four steps:

  • Access the session and start output buffering,
  • instantiate an object which controls information gathering,
  • fetch information from the $_SESSION array,
  • encode and send the JSON object to the Ajax client

PHP program “check_progress.php5”

// start session and output buffer
session_start();
ob_start(); 

// Instantiate progress checker object 
$PC = new ProgressChecker();  

// retrieve and send information 
$PC->check_progr(); 
$PC->encodeAndSendJson(); 
exit; 

// ------------------------------
class ProgressChecker  
{
	//Some variables 
	...
	function __construct() {
		....
	}
	function check_progr() {
		....
	}
	function encodeAndSendJson() {
		....
	}
}

Why did we need the output buffering ob_start()? By using the buffer we control unwanted output produced e.g. by warnings of the PHP interpreter! We want to guarantee that our Ajax client really receives a JSON object (which it expects!) and nothing else. See the third article of this series for more remarks on this
point.

Remark, 05.07.2015: We have corrected a mistake in the originally published code. “ob_end_clean()” was removed from the main program code. Note that “ob_end_clean()” must be performed just before the Ajax response is sent. This happens in the method encodeAndSendJson(); see below.

Although not required we put all functionality into a class. This will give us more flexibility in case we need to extend the functionality of the object for progress tracking later on.

Some variables of the class “ProgressChecker”

In our class we first set some initial values of elements of an array “$ajax_response[]“. This array will later on be encoded as the JSON response object of our Ajax transaction.

	// The array to build the JSON object from 
	var $ajax_response = array();
	// Some useful variables 
	var $percentage = -1;
	// Error handling 
	var $err = 0;
	// variables to compose the key of progress information in $_SESSION
	var $sess_key_progress 	= '';
	var $key_POST = "progress_key";
	
	function __construct() {
		
		$this->ajax_response['sess_key_progress'] = '';
		
		$this->ajax_response['msg'] = '';
		$this->ajax_response['err'] = 0;
		$this->ajax_response['err_msg'] = '';
		$this->ajax_response['sys_msg'] = '';
		
		$this->ajax_response['finalized'] = 0;
		$this->ajax_response['time'] = '';
		$this->ajax_response['secs'] = 0.0;
		$this->ajax_response['diff_secs'] = 0.0;
		$this->ajax_response['end_time'] = '';
		$this->ajax_response['end_secs'] = 0.0;
	
		$this->ajax_response['bytes'] = -1;
		$this->ajax_response['diff_bytes'] = -1;
		$this->ajax_response['total'] = -1;
		$this->ajax_response['percentage'] = -1;
		$this->ajax_response['rate'] = -1.0;
	
		$this->ajax_response['fs_limit'] = -1;
		$this->ajax_response['max_inp_time'] = -1;
	}

 
We organize the information that may be required by the client: the composed key to access progress information in $_SESSION, some time information and real progress information.

“bytes” represents the number of Bytes received so far on the server, “total” is the total number of bytes of the transferred file (i.e. its size), “percentage” is calculated from “bytes” and “total”. In addition we deliver a measured data transfer “rate” in [Bytes/msec]. The reader may compare this with the result handling on the Javascript client side described in the last article of this series.

Last but not least we initialize some variables which shall deliver useful information about parameters set in the “php.ini” file on the server.

You see that we intentionally set some variables to negative values. Such an initialization can help to analyze potential errors on the server and especially after some response has been returned on the client.

Reading progress information from the $_SESSION array

The core of our class is the method check_progress():

function check_progr() {
	
	// Compose the key for the information in the $_SESSION array  
	if (!isset( $_POST[$this->key_POST] ) ) {
		$this->err++;
		$this->ajax_response['err'] = 1;
		$this->ajax_response['err_msg'] .= '<br>The required key was not delivered in $_POST';
	}
		
	if ( $this->err == 0 ) {
		$this->sess_key_progress .= ini_get("session.upload_progress.prefix"). $_POST[$this->key_POST];
		$this->ajax_
response['sess_key_progress'] = $this->sess_key_progress;
	}
		
	// Check for progr. information 
		// We have set session.upload_progress.cleanup = Off
		// So we do have some information here
			
	if (!isset($_SESSION[$this->sess_key_progress]) || empty($_SESSION[$this->sess_key_progress])) {
		$this->err++;
		$this->ajax_response['err'] = 2;
		$this->ajax_response['err_msg'] .= '<br>Error: The progress information could not be read from $_SESSION';
		return; 
	}
			
	// Save some information at the first poll for later rate calculations 	
	if (!isset($_SESSION['first_secs'])) {
		$_SESSION['first_secs'] = microtime(true);
	}
	if (!isset($_SESSION['first_bytes'])) {
		$_SESSION['first_bytes'] = $_SESSION[$this->sess_key_progress]["bytes_processed"];
	}
		
	// Get some php.ini parameters
	$file_size_limit = ini_get("upload_max_filesize");
	$this->ajax_response['fs_limit'] = $file_size_limit; 
	$this->ajax_response['max_inp_time'] = ini_get("max_input_time"); 

	// investigate whether file size is too big and return error 
	$file_size = $total / 1024 / 1024; 
	if ( $file_size > $file_size_limit ) {
		$this->err++; 
		$this->ajax_response['err'] = 3;
		$this->ajax_response['err_msg'] .= "<br>Error: The file has a size of " . number_format($file_size, 2, '.', '') . " MByte and is bigger than the allowed limit on the server (" . $file_size_limit . " MByte)"; 
		return; 	
	}
	
	// Retrieve progress information 
			
	// Bytes
	$current = $_SESSION[$this->sess_key_progress]["bytes_processed"];
	$total 	 = $_SESSION[$this->sess_key_progress]["content_length"];
	
	$this->ajax_response['percentage'] = ($current < $total) ? ($current / $total * 100) : 100;
	$this->ajax_response['bytes'] = $current;
	$this->ajax_response['total'] = $total;
		
	$this->ajax_response['time'] = date('%d.%m.%Y-%H:%M:%S');
	$this->ajax_response['secs'] = microtime(true);
	$this->ajax_response['first_bytes'] = $_SESSION['first_bytes'];
	$this->ajax_response['first_secs'] = $_SESSION['first_secs'];
	$this->ajax_response['diff_secs'] = $this->ajax_response['secs']  - $_SESSION['first_secs'];
	$this->ajax_response['diff_bytes'] = $this->ajax_response['bytes'] - $_SESSION['first_bytes'];
	
	// Rate in bytes / msec
	if ($this->ajax_response['diff_secs'] > 1.0 ) {
		$this->ajax_response['rate'] = floor($this->ajax_response['diff_bytes'] / ( $this->ajax_response['diff_secs'] * 1000.0));
	}
		
	// upload finalized ? if yes => cleanup of $_SESSION 
	$done_upl 	= $_SESSION[$this->sess_key_progress]["done"];
	$done_file 	= $_SESSION[$this->sess_key_progress]["files"][0]["done"];
		
	if ($done_upl && $done_file) {
		$this->ajax_response['end_time'] = $this->ajax_response['time'];
		$this->ajax_response['end_secs'] = $this->ajax_response['secs'];
	
		// cleanup of progress variables
		// the real session will be closed by the upload job
		unset($_SESSION[$this->sess_key_progress]);
		unset($_SESSION['first_secs']);
		unset($_SESSION['first_bytes']);
	}
		
}

 
As we already know, we first have to compose the key to access progress information in the $_SESSION array. We must use

  • both information provided by the client to uniquely identify the file transfer process to which our polling job refers
  • and some specific information of the “php.ini” file.

See the third article of this series for more information about this point. Remember that we explicitly took care about the fact that key relevant information reaches the server first – i.e. before the file data.

In the block of program statements we obviously test
the existence of progress data – and stop our activities with setting an error flag and an error message in our response array. Such an error situation must be recognized and handled on the client side; see the last article of this series.

Then we add some initial time information and byte information to new elements of the $_SESSION array. Obviously, this happens only at the first polling job. This stored information will later on be used for rate estimations.

Important note:

Our simplified approach to rate estimation via additional $_SESSION variables will only work if you trigger exactly one upload process at a time. If you triggered several upload jobs in parallel (i.e. before previously started jobs have finalized and within one and the same session) you would need to make all progress information which you add to $_SESSION dependent on the identifier of the transfer process.

We leave this extension to the reader and assume that different upload jobs are started from the client only sequentially.

We now fetch 2 parameter values from the “php.ini” file: for the maximum allowed size of upload files and the maximum allowed input parsing time. The latter value determines how much time can be used on the server to deal with the transferred (POST) data. This is a critical limit about which the client should be informed.

In addition: If we find that the size of the transferred file is too big, we return an error to the client.

Next, we retrieve progress information, calculate the reached percentage of transferred bytes and estimate a transfer rate. To avoid any problems with divisions by zero we deliver a rate value only after at least a second (i.e. the default the update time interval on the server for progress information) has passed. Remember that these rate values are used on the client side to adapt the polling period to reasonable values.

Finally, we check by some criteria whether the transfer job has already finished and all file data are received. If this is the case, we erase progress related information from the $_SESSION array. This is absolutely necessary if you want to start further upload jobs during the same PHP session.

Important note:

We are only able to retrieve the information about the end of the file transfer if the progress information is not automatically erased from the $_SESSION array by the tracking engine of the server itself. Otherwise our last polling job would almost always run into problems. So, for our Ajax controlled polling to work flawlessly, we must set:
session.upload_progress.cleanup = Off
in our “php.ini”-file – and do some cleanup ourselves.

At this point I also want to remind the reader that our main job for dealing with the uploaded file data may already have started before the last polling job is triggered. Due to the length of polling time interval such a situation is very probable. We had to take care of this situation on the Javascript client side to avoid a confusing mixture or overwriting of returning messages from both PHP jobs; see the last article of this series for more remarks on this point.

Encoding the JSON object

The final step is trivial:

function encodeAndSendJson() {
	$this->ajax_response['sys_msg'] .= ob_get_contents(); 
	ob_end_clean();
	$response = json_encode($this->ajax_response);
	echo $response; 
}

Some example output of the Ajax controlled client/server interaction for the file upload

After all this theory let us now have a brief look of output of the
Ajax interaction between the client and the server. Initially we give the user a chance to select a file

upl_form_1
 
The PHP job that created the web page which contains our upload form also started the required PHP session. You may ignore the upper parts of the displayed excerpt of our web page. The interesting things happen in the lower form which allows for file selection.

The attached information elements (progress bar, small text areas) below actually belong to a separate and different web page area (DIV container) which contains an invisible form for the control of the progress polling jobs.

Note that in the displayed test case we have chosen a ZIP file container (which actually contains 5 files with 5 million records each). Then we start the Ajax controlled transfer with a click on the button “Start Upload”. This job starts the file transfer and the first Ajax transaction. This will trigger the main PHP job “handle_uploaded_init_files.php5” for file processing when the data transfer has finalized.

upl_form_2

In parallel the sequence of polling jobs is automatically started in the background. Each Ajax transaction in the time loop triggers a job “check_progress.php5” on the server, which retrieves progress information from $_SESSION as discussed above. The information of each PHP polling job returned to the client via a JSON object is used to enlarge the progress bar and to display additional information text about the progress.

upl_form_3
upl_form_4

Note the display of the bytes transferred, the time elapsed and the remaining time estimated from the measured rate.

After all file data have been received at the server we display some final information in green on the left side. After that the main PHP job (triggered by our original form submit) takes over and the status bar runs again – this time indicating the processing of the individual files we moved to the server in our Zip file container.

upl_form_6

We turn to the handling of the ZIP container and its contents in one of the next articles of this series.

Adaption of the polling interval

In our previous example the polling period stayed at the lowest boundary of 1200 msec. The file (around 80 MByte) was not big enough for an adjustment at the given rate. However, a different example with a bigger file around 150 MByte shows the adaption of the polling period quite clearly. Excerpts from respective messages in the console.log:

time = 1433754422037, 
polling_
period = 1200

Present upload percentage = 3.0057897183539
rate is 1485, poll_soll = 1501,
polling_period = 1440

Present upload percentage = 5.0072492005327
rate is 1341, poll_soll = 1652  
polling_period = 1652

Present upload percentage = 7.0087017625639
rate is 1235, poll_soll = 1785
polling_period = 1652

Present upload percentage = 8.0094300619559
rate is 1209, poll_soll = 1821
polling_period = 1821

Present upload percentage = 11.011615536811
rate is 1250, poll_soll = 1765
polling_period = 1821


Present upload percentage = 12.012343836203
rate is 1219, poll_soll = 1807
polling_period = 1821

Present upload percentage = 13.013072712274
rate is 1193, poll_soll = 1844
polling_period = 1821

Present upload percentage = 14.013802741702
rate is 1172, poll_soll = 1875
polling_period = 1821

Present upload percentage = 16.015263377239
rate is 1231, poll_soll = 1790 
polling_period = 1821

Present upload percentage = 17.01599225331
rate is 1211, poll_soll = 1818
polling_period = 1821

Present upload percentage = 18.016722859418
rate is 1193, poll_soll = 1844
polling_period = 1821

Present upload percentage = 19.017454042205
rate is 1177, poll_soll = 1868
polling_period = 1821

Present upload percentage = 21.018911217668
rate is 1206, poll_soll = 1825
polling_period = 1821

 
The rate changes from 1200 msec to 1821 msec in 4 steps. That is exactly what we wanted to achieve.

However, there is a glitch of one additional percent every 4th step. This can be explained by the fact that our polling interval allows for around 1.2 % effective change but that we pick up only a 1% change due to the servers settings between 2 polling events. As the update period on the server is a bit smaller and thus a bit asynchronous to our polling events a glitch is bound to happen around every 4th event on the server. The reader may sketch the polling events and boundaries of the update period on the server in a drawing to get a clear understanding. We can ignore the glitch for all practical purposes.

Note that for larger files at the same rate the adaption would take more steps – but the behavior would more or less be the same.

A first conclusion

So far, so good. Obviously, it is possible to use the progress tracking of PHP versions ≥ 5.4 in combination with a series of Ajax controlled polling jobs. The Javascript side was a bit tricky – but we got it working – and we now have full control about the upload process and its messages. At least in principle …

However, there are still some severe issues. Especially, if the upload file is big and the rate is small. Then our mechanism may run into trouble due to parameter settings of the PHP engine. In the next article of this series

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – VI

we will therefore discuss how we can get some control over situations in which violations of server limits become probable.

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – IV

In the last article of this series

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – III
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I

we saw that PHP 5.4 may write progress information about file uploads into special elements of the $_SESSION-array on the server. We discussed that the PHP session must already be established before the upload process is started by the web client via an Ajax process.

Furthermore, we have understood the following: The PHP target program which is called by the original Ajax request – i.e. the request that starts the file upload – will not be of any use for sending progress information about the file transfer to the client: The code execution of this PHP program only starts after all POST data, including all file data, have completely been transferred to the server.

So, how do we get the progress information which the server continuously updates in the PHP $_SESSION array from the server to our web client during the upload process? In this article we look at a solution based upon a general polling scheme and respective prerequisites on the client, i.e. on the Javascript side. As we want to transfer the status information in the background we again talk about Ajax controlled jobs.

A sequence of Ajax polling jobs

The situation is very much the same as for long lasting PHP jobs [server RUNs] whose status shall be followed by an Ajax mechanism. This can be solved by starting a sequence of periodic Ajax jobs from the same web page from which the original PHP RUN was started. See e.g. the article
Fallen beim Statuscheck lang laufender PHP-Jobs mit Ajax – IV
in this blog and the following drawing:

polling

These periodic jobs “poll” the progress information from some storage on the server – in our case from the $_SESSION array. Therefore, we call the periodic jobs “progress polling jobs”.

A status monitoring with polling jobs has advantages and disadvantages. One advantage is that it does not depend on special HTML 5 properties or a special jQuery plugin. Another advantage is that we get complete control over what data are exchanged between server and client. One disadvantage is – as we shall see – that we may need to dynamically adapt the polling time interval to the measured rate of the file transfer. A further potential disadvantage is that it may become difficult to react to problems occurring during the ongoing transfer of the file data. We shall have to come back to this point in a later article.

However, in the present article we ignore potential problems and sketch the basic outline of the polling control.

A form for sending information together with each Ajax polling request

On the web page we associate a small form with the polling jobs. This form is
added in a special separate “progress” DIV contaimer of our Template [TPL] for the web page. See: CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II

	<!-- A div to display the upload progress -->
	<div id="progress">
		<form id="form_upl_progr" style="height:0;" action="check_progress.php5" method="POST">
			<input type="hidden" name="progr_key_name" id="hinp_upl_progress_key_name" value="upl">
		</form> 
		<div id="progress_bar_div">
			<div class="outer_bar_div">
				<div id="bar"></div>
			</div>
		</div>
		<div id="progr_msg_cont">
			<div id="progr_msg_bg"></div>
			<div id="progr_msg_div"><p id="progr_msg"></p></div>
		</div>
		<div id="trf_msg_cont">
			<div id="trf_msg_bg"></div>
			<div id="trf_msg_div"><p id="trf_msg"></p></div>
		</div>
		<div id="imp_msg_cont">
			<div id="imp_msg_bg"></div>
			<div id="imp_msg_div"><p id="imp_msg"></p></div>
		</div>
		.....
		.....
	</div>

 
Obviously, we have named the PHP target program collecting status data “check_progress.php5“. In our very basic example the form’s only contents is a hidden input field whose value is the identifier for the upload process.

As we have seen in the last article this identifier information is required to compose the key for accessing progress information in the $_SESSION array. Note that the identifier value is identical to the one of the corresponding hidden input field in the original upload form we discussed in our last article. (In a productive example both these values should be set by the TCO (template controller object) that uses the TPL to create the web page.)

Note: In a real world example it may be necessary to send more information to the server – especially for error or problem handling. Our simple form can easily be extended by more variables to be sent to the server.

In the HTML code of our DIV we have already included some elements whose purpose later on will be to display specific information about the file transfer progress and also the following import of files into the database. The DIV with the id=”bar” will simulate a growing progress bar. For those who prefer a picture – we intend to display something like this:

upl_form

Our “progress” DIV encapsulates the elements of the lower part. There is no need to discuss CSS properties here.

A Javascript CtrlO for the “Progress Information Area” on the web page

We follow our overall Javascript policy to create a “Control object” – a CtrlO – for the “progress” DIV which encapsulates all event handling for the HTML elements plus all Ajax communication handling. We call the correspondent constructor function “Ctrl_Upl_Progr()“.

This CtrlO shall be created as a typical singleton and as part of a GOC object (= Global Object Controller) – i.e. its handle or reference will be assigned to some variable

window.GOC.Obj_Upl_Progr

when the web page is loaded. The GOC may also dispatch information about the existence and the address of the object “GOC.Obj_Upl_
Progr” to other CtrlOs. Without going into details we indicate “dispatching” by a method “GOC.push_object_references()” :

Excerpts from Javascript code of a JS file loaded in the <header>-tag of the upload web page


// Function called by the web page at the onLoad event
function init() { 
	...
	// The CtrlO object for the DIV displaying a form to upload files    
	GOC.Obj_File_Upl = new Ctrl_File_Upl('File_Upl'); 
	
	// The CtrlO object for the Control of the DIV with progress information 
	GOC.Obj_Upl_Progr = new Ctrl_Upl_Progr('Upl_Progr'); 
	...
	...
	// Dispatch references of all other CtrlOs to each of the CtrlOs 
	GOC.push_object_references(); 
}
....
.....
// Constructor function 
function Ctrl_Upl_Progr(my_name) {
       	
	// Required only for possible references to other CtrlOs 
	this.GOC = GOC;
	// The singleton object in the GOC  
	this.obj_name = "Obj_" + my_name; 
        
	// Selectors
	this.id_progr_form	= "#form_upl_progr"; 
	this.id_progr_div 	= "#progr_bar_div"; 
	this.id_bar 		= "#bar"; 
	this.progr_msg_cont 	= "#progr_msg_cont"; 
	this.progr_msg_p 	= "#progr_msg"; 
	this.trf_msg_cont	= '#trf_msg_cont";
	this.trf_msg		= '#trf_msg";
          
	// Register some events  
	this.register_events(); 

	// timer and initial interval for retrieving progress information 
	this.upl_min_polling_interval = 1200; 
 	this.upl_polling_interval = 1200; 
	this.upl_max_polling_interval = 2000; 
	this.upl_progr_timer = null; 
	this.upl_progr_count = 0; 
	this.upl_progr_count_limit = 100; 
           
	// did the last poll job return ?   	
	this.poll_return = 1; 

	// some progress information 
	this.file_size 	= 0;
	this.last_upl_percentage = 0; 
	this.present_upl_percentage = 0; 
       
	// rate of bytes per msec
	this.present_rate = -1.0; 
	this.diff_bytes	= 0.0; 
	this.diff_secs = 0.0; 
	this.first_bytes = 0.0; 
	this.first_secs = 0.0; 
	this.bytes = 0; 
      
	// start time 
	this.start_time = Date.now(); 
 
	// deal with race condition between messages from the last polling job 
	// and response messages from the executing main PHP job   
	this.race_finalized = 0;

	// Parameters for the Ajax connections 
	this.ajax_url = $(this.id_progr_form).attr('action'); 
	this.ajax_return_data_type = 'json';
 
	// Ajax return messages
	this.err = 0; 
	this.msg = ''; 	
	this.err_msg = ''; 	
	this.sys_msg = ''; 	
}   
....
....
/* Container for global objects - the "this"-operator refers here to the Global object - becoming "window" in browsers */ 
this.GOC = new Global_Objects();
  

 
For the meaning and purpose of the GOC see also a comment to the second article of our series.

The CtrlO “Obj_File_Upl” for the upload form should already be familiar from a previous article. Although one may dispute the approach of assuming and using “singleton” objects in the context of other Javascript tasks, I find the CtrlO approach very clear and convenient for dealing with well defined web page areas providing a certain service: The reason is that such UI areas are relatively individual; so they should be controlled by equally individual CtrlOs. Nevertheless we still try to obey two rules to avoid any dependency on the “singleton” property :

  • The CtrlO methods should not require any special knowledge about the internal structure of the GOC.
  • The CtrlO methods should still work even if several CtrlO objects of the same type are created (full encapsulation).

The latter may become relevant later; e.g. if you want to upload several distinct files for different purposes on the same web page.

Note that we
call a method “register_events()” to register methods for handling events occurring for elements of our DIV container. This is a central element of a CtrlO.

Polling frequency and progress update period on the server

Note that we set the initial polling time interval on the client to be 1200 msec. This means that (initially) every 1200 msec an Ajax request will be sent to the server. OK – but this raises the question:

How often is the progress information in the $_SESSION array updated on the server? What is the update frequency on the server compared with our polling frequency?

Not surprisingly, the progress update period on the server is defined by some parameters in the file “/etc/php5/apache2/php.ini“:

; How frequently the upload progress should be updated.
; Given either in percentages (per-file), or in bytes
; Default Value: "1%"
session.upload_progress.freq =  "1%"

; The minimum delay between updates, in seconds
; Default Value: 1
session.upload_progress.min_freq = "1"

 
The latter value min_freq has to be an integer. Please, keep in mind that some of the logic we later on implement in our CtrlO only makes sense only if the polling interval on the client is always chosen to be somewhat longer than min_freq on the server.

Now we consider an important question about the whole approach: Can a constant polling interval lead to trouble?

Yes, this is possible. Think about the following situation:

If our connection and the data transfer are very slow we may not see any transfer progress for some polling intervals due to the consequences of the php.ini parameters named above. The progress values may not change although file data arrive continuously – but slowly – at the server.

This may give us a false impression of what is going on and it may impact program decisions whether to stop the polling due to a (falsely) detected potential error situation. In case of a small data transfer rate we obviously should use a relatively long polling time interval to see progress between two consecutive polls. Already this consideration indicates that we need an adaptive polling time interval. Furthermore, we also need to deal with situations where the server responds slowly – i.e. with a period longer than our polling interval. A reasonable approach in such a case is that we should only send a new polling job to the server if we already have received a response to our last polling job. If this is not possible with the chosen polling interval we have to change it. We shall come back to both points below.

Methods of our new CtrlO

What methods does our CtrlO need to control the sequence of Ajax jobs? According to our drawing we need to trigger a time loop. A proper Javascript function for this purpose would be “setInterval()“. However, as the context of the function “setInterval()” is the global object (i.e. the browser window) we have to be careful about how we call the function/method

get_status_by_ajax()

that we want to execute periodically :

// method to initiate the time loop for progress control 
Ctrl_Upl_Progr.prototype.get_status_by_ajax = function(setStartTime) {
	
    	if (typeof setStartTime === "undefined" ) {
    		setStartTime = 0; 
    	}
	if ( setStartTime == 1) {
		this.start_time = Date.now(); 
	}
	this.upl_progr_timer = setInterval(this.progr_submit.bind(this),
 this.upl_polling_interval); 
	if ( this.upl_status_timer != null ) {
		console.log ("upl_progr_timer is set !"); 
	}
    	
	// cleaning the message area of our web page
    	if ( setStartTime == 1  ) { 
    		this.SO_Msg.show_msg(0, '');
    	}
}; 

// Method to trigger submit events for progress polling jobs   
Ctrl_Upl_Progr.prototype.progr_submit = function() {
	$(this.id_progr_form).submit();  
};

// Register events and delegate responsibility to local methods   
Ctrl_Upl_Progr.prototype.register_events = function() {
	$(this.id_progr_form).submit( 
		jQuery.proxy(this, 'poll_progress') 
	);
};

 
The method triggered periodically is obviously one of our CtrlO object – “progr_submit“. This method triggers the submit event for our new form which in turn is registered with a CtrlO callback method “poll_progress“. To delegate the responsibility for the submit event to a local method of the CtrlO we used jQuery’s proxy mechanism. As we used $.proxy() in a previous article of this series we are already familiar with this approach to encapsulate all control mechanisms inside the local CtrlO object.

In addition we introduced a parameter “setStartTime” that distinguishes between the start of a file upload and later calls of the method “get_status_by_ajax(setStartTime)“.

Invocation of a local CtrlO method in setInterval()

A closer look shows that we called the “progr_submit()” method in a special way: Why do we need to attach the “.bind(this)” method to the callback function?

As so often with Javascript and jQuery the question behind this “trick” is what the context of the “this” operator in an objects method will be, when this method is indirectly called as a callback. Naturally, we want the “this” operator to point to our CtrlO, because we want to encapsulate everything there. But as a matter of fact the “this” operator of the methods/functions called by setInterval() will point to the global context! But even this depends a bit on how exactly you call the method:

A call as

this.upl_progr_timer = setInterval(this.progr_submit, this.upl_polling_interval);

would not work for us because the “this” context would point to the global object when the callback is invoked.

A call as

this.upl_progr_timer = setInterval(this.progr_submit(), this.upl_polling_interval);

would work – but only once. Here the context really refers to that of the outside calling function “get_status_by_ajax()”. But this context of the calling function is destroyed directly after the first call. Working around this specific problem would require a closure – see:
http://stackoverflow.com/questions/10944004/how-to-pass-this-to-window-setinterval
http://stackoverflow.com/questions/2749244/javascript-setinterval-and-this-solution
Both closure approaches discussed in the named articles are nice – but actually not necessary for present browsers.

Actually, there is one special way of enforcing the context of the “Obj_Upl_Progr” correctly via using the object’s address in the global context explicitly:

this.upl_progr_timer = setInterval(“GOC.Obj_Upl_Progr.progr_submit()”, this.upl_polling_interval);

Note that the brackets “()” after the method name are required in this type of approach (at least in FF)!

However, you may set a big question mark behind such a program design and its implied consequences because explicit knowledge about the usage of the “Upl_Progr”
class in the GOC is required and the whole approach will only work with singletons. Nevertheless: Our overall design of CtrlOs naturally deals with singletons and a GOC dispatcher – so this solution is somewhat noteworthy. It violates, however, our 2 rules set above.

So, if you want to stay clean use “bind(this)” as in our suggested solution:

this.upl_progr_timer = setInterval(this.progr_submit.bind(this), this.upl_polling_interval);

“bind(context)” sets the context explicitly to the local object from where setInterval is called and will even work in IE > 9. Note that in this approach we do not need to invoke any specific global knowledge about the GOC and that it is not restricted to singletons!

Where from do we start the time loop for polling?

OK, we now have a method to start the time loop for polling. But, we have not yet answered the question, from where and when we start this method. This must of course be done directly after the main job for the file upload has been started. We remember that in
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II
we started the main job for file uploading and processing by a special method of the responsible CtrlO “Ctrl_File_Upl”. We extend the code now a bit:

Ctrl_File_Upl.prototype.upl_file = function(e) { 
	....
	....
	// Ajax submit  
		
	// URL 
	console.log('upl_file :: url = ' + this.url);
	url = this.url + "?file";
	console.log('upl_file :: submitted url = ' + url);
			
	// Time 
	this.date_start = new Date(); 
	this.ajax_transfer_start = this.date_start.getTime(); 
			
	// Setup and submit Ajax 
	$.ajax({
		// contentType: "application/x-www-form-urlencoded; charset=ISO-8859-1",
		url: url, 
		context:  GOC[this.obj_name],
		timeout: this.timeout,
		data: form_data, 
		type: 'POST', 
		cache: false, 
		dataType: 'json', 
		contentType: false,
		processData: false, 
	
		error: this.error_ajax_file_upl,
		success: this.success_ajax_file_upl
	});
			
	// Start the progress polling 
	// --------------------------	
        this.SO_Upl_Progr.race_finalized = 0; 
        this.time_start = Date.now(); 
        this.SO_Upl_Progr.get_status_by_ajax(1); 
			
};	

 
We just addes some statements that refer to our new CtrlO
“Ctrl_Upl_Progr”. Our GOC dispatcher has made this CtrlO available locally in the CtrlO “Ctrl_File_Upl” via the variable
“this.SO_Upl_Progr”. Hence:

this.SO_Upl_Progr.get_status_by_ajax(1);

The other statements set values to other variables used in the CtrlO “Ctrl_File_Upl” later on.

Ajax Polling Requests

As you already may have expected we now set up and trigger an Ajax request with the method poll_progress(). As our form is fairly simple, we can just serialize the form data and e.g. use the $.post variant of jQuery’s Ajax interface:

Ctrl_Upl_Progr.prototype.poll_progress = function(e) {
         
	//always count polling steps  	
	this.upl_progr_count++; 
        
	// prevent default action of the submit event 
 	e.preventDefault();
    	
	// check against a maximum of periodic calls
 	if ( this.upl_progr_count > this.upl_progr_count_limit ) {
 		clearInterval(this.upl_progr_timer);
 		
console.log("progress timer eliminated"); 
 		this.err_msg += "<br><br>Progress polling was stopped due to max request number reached!"; 
 		this.display_msgs(); 
 		return 1;
 	}
        
	// Increase the polling interval if poll_return is not 1 
 	// No new polling job is started if the last did not yet return 
	if (this.poll_return == 0 ) {
		// Adjust interval 
		if (this.upl_polling_interval < this.upl_max_polling_interval ) {
			clearInterval(this.upl_progr_timer);
			this.upl_polling_interval += 100;
			// delay a bit
			setTimeout(this.get_status_by_ajax.bind(this), 100, 0); 
		}
		// jump over one execution 
		else {
			return 2; 
		}
	}
    		
	// start next poll job
	else {
		console.log("Upl_Progr object - poll_progress() :: ajax url = " + this.ajax_url);
		var form_data = $(this.id_progr_form).serialize(); 
		$.ajaxSetup({
			context:  this
		});
		$.post(this.ajax_url, form_data, this.progress_response, this.ajax_return_data_type);  
		this.poll_return = 0; 
		console.log("Upl_Progr object :: submitted Ajax progress polling query to server" );
	}	
}; 

 
Note that we set the context of the Ajax interface explicitly! Thus, we guarantee that the method

this.progress_response

which deals with the JSON answer to our request gets our CtrlO as the context – meaning that the “this” operator will point to our CtrlO and not to some HTML element. We like to emphasize:

Not setting the context for the “this” operator is a common source of errors and mistakes when using callback methods to handle status events or responses in course of Ajax transactions.

Of course, we could have included all parametrization and the call of the Ajax target program in one statement as in our last article. The usage of $.post() in our example has no other reason than showing and discussing an alternative. The parameter for the type of the return data had already been set to “json” in the constructor function.

Remark 28.07.2105:

Due to a mail of a reader I like to point out that using a combination of $.ajaxSetup() with $.post() may have disadvantages because $.ajaxSetup() sets standards for all coming Ajax transactions. Certainly, you do not want the context of all future Ajax transactions point to “Ctrl_Upl_Progr”. So, you would have to take care to reset this value as required later on. If you absolutely want to use $.post(), a much more intelligent solution would therefore be to use it in the form “jQuery.post([settings])” with “settings” representing an Ajax key/value parametrization object. See http://api.jquery.com/jquery.post/#jQuery-post-settings. Or just keep using $.ajax() here, too.

Counting the number of executing “progress_poll()” is going on unconditionally until a maximum of allowed calls is reached. This helps to avoid an unlimited time loop. Note in addition that we do not submit a new job if the last one did not yet return some answer. If we experience such a situation we adapt by extending our polling interval slowly but systematically until a maximum value is reached. Thereby, we react adaptively to slow servers or slow connections.

Now you may ask: What happens if the upload progress is going on slowly and the maximum allowed numbers of polling calls is too small? Good point! We will react to this by changing the maximum allowed number of poll calls when dealing with the answers of the server – but only if some upload progress is still measurable. See below.

Dealing with the JSON response of the polling jobs
from the server

Now, let us do something about the Ajax response. We expect some element of the JSON object to deliver information about the total size of the file whose upload is ongoing in the background and some information about the percentage of data already transferred to the server. As a simple variant we suggest a method similar to the following code:

Ctrl_Upl_Progr.prototype.progress_response = function(json_progr_result, success_code, jqxhr) {
        
	var msg, err_msg, sys_msg, end_time, count; 
	var poll_soll, poll_diff, poll_rel, time_remain, time_remain_str, time_spent, time_spent_str, time_msg; 
	var div, div_long, p_float_stop, span, span_w; 
	var fs, fs_str, by, upl_bytes, upl_bytes_str, upl_by; 

	// Response  status 
	console.log("Progress response :: received Ajax JSON object from server" );                
	var status = jqxhr.status; 
	var status_txt = jqxhr.statusText; 

	// polling can be continued         
	this.poll_return = 1;     
        
	// error detected by the server ?
	this.err = json_progr_result.err;
	if ( this.err > 0 ) {
		.....
		// general error treatment 
		.....
	}

	// different message types contained in the JSON response  
	this.msg = json_progr_result.msg; 
	this.err_msg = json_progr_result.err_msg; 
	this.sys_msg = json_progr_result.sys_msg; 

	// progress information delivered by the server  
	this.present_upl_percentage = json_progr_result.percentage;
	this.present_rate = json_progr_result.rate; 
	this.file_size = json_progr_result.total;
            
	// Current number of uploaded bytes 
	this.bytes = json_progr_result.bytes; 
	var upl_bytes 	= this.bytes / 1024.0; 
	var upl_by 	= " KB"; 
	if ( upl_bytes > 1024 ) {
		upl_bytes = this.bytes/(1024 * 1024);
		upl_by = " MB"; 
	}
	upl_bytes = upl_bytes.toFixed(2); 
	upl_bytes_str = upl_bytes + upl_by;
		        
	// file_size 
	fs = this.file_size/1024.0;
	by = " KB"; 
	if ( fs > 1024 ) {
		fs = this.file_size/(1024 * 1024);
		by = " MB"; 
	}
	fs = fs.toFixed(2); 
	fs_str = fs + by; 

	// Checking for upload finalization
	end_time  = json_progr_result.end_time; 

	// Shall the associated time loop be stopped ? 
	if ( end_time.length > 0 || this.present_upl_percentage == 100 ) {
		clearInterval(this.upl_progr_timer); 
		console.log("progress timer stopped regularly do to finished upload process on the server");
		// this.present_upl_percentage = 100; 
		this.msg += "<br>File transfer to the server is complete !<br>"; 
	}
        	
	// Set the limit up if the max number is reached but upload is still processing regularly 
	else {
		if (this.present_upl_percentage > 0 && this.present_upl_percentage > this.last_upl_percentage 
		&& ( this.upl_progr_count + 2 > this.upl_progr_count_limit	) ) {
			this.upl_progr_count_limit += 100; 
			console.log("limit for polling counts raised by 100!" ); 
		}
	}
            
	// Display information about percentage - e.g. in form of a progress bar 
	if (this.present_upl_percentage > 0 ) {
		this.last_upl_percentage = this.present_upl_percentage; 
		this.display_upl_percentage(); 
		if ( this.last_upl_percentage < 100 ) {
			this.msg += "<br>File upload is progressing!<br>" 
		}
		this.msg += this.last_upl_percentage.toFixed(2) + " percent of the file have been transferred"; 

		// Display info in special field 
		var msg_trf = "Transfer: " + upl_bytes_str + " of " + fs_str + " :: " + this.last_upl_percentage.toFixed(2) + " %"; 
		$(this.id_progr_msg_p).html(msg_trf);
	}

	// concatenate upload messages and display them 	
	if ( this.err_msg != '' ) {
	       	this.msg += "<br><br>" + "<span style=\"color:#
A90000;\">" + this.err_msg + "</span>"; 
	}
	if ( this.sys_msg != '' ) {
		this.msg += "<br><br>" + this.sys_msg; 
	}
	// Only display messages in a special message area if you do not overwrite
	// messages coming already from the main PHP job or if an error occurred   
	if (this.race_finalized == 0 || this.err > 0  ) { 
		this.display_msgs();
	}
 
        // switch of setInterval's timer in case of error signals from the server 
       	if ( this.err > 0 ) {
       		// For all of the following error types 
       		if ( 1 <= this.err <= 3 )
        		clearInterval(this.upl_progr_timer);
        	}
	}
           
	// Adaption of the polling interval and time estimates 
	if ( this.err == 0 && this.present_upl_percentage > 3 && this.present_rate > 0 ) {
		poll_soll = 1.2 * this.file_size / 100.0 / this.present_rate; 
		poll_soll = Math.round(poll_soll) + 100; 
		poll_diff = poll_soll - this.upl_polling_interval; 
		poll_rel = ( Math.abs(poll_soll - this.upl_polling_interval) ) / this.upl_polling_interval  
     		
		// Only do something if the deviation is bigger than a limit 
		if ( poll_rel > 0.10 ) {
			// Limit the relative change 
			if ( poll_rel > 0.2 ) {
				if (poll_diff < 0 ) {
					poll_soll = 0.8 * this.upl_polling_interval;  
				}
				else {
					poll_soll = 1.2 * this.upl_polling_interval;  
				}
			}
			else {
				if (poll_diff < 0 ) {
					poll_soll = (1.0 - poll_rel) * this.upl_polling_interval;  
				}
				else {
					poll_soll = (1.0 + poll_rel) * this.upl_polling_interval;  
				}
			}
	        			
			if ( poll_soll > this.upl_min_polling_interval) {
				this.upl_polling_interval = poll_soll; 
				if (poll_soll > this.upl_max_polling_interval ) {
					this.upl_max_polling_interval = poll_soll + 100; 
				}
				clearInterval(this.upl_progr_timer);
				// delay a bit
				setTimeout(this.get_status_by_ajax.bind(this), poll_soll, 0); 
			}
		}
	        		
		// time estimate
		time_remain 	= ( (this.file_size - this.bytes) / this.present_rate) / 1000.0;
		time_remain_str = time_remain.toFixed(1) + "s"; 
	        		
		// time spent 
		time_spent 	= (Date.now() - this.start_time); 
		time_spent_str 	= (time_spent / 1000.0).toFixed(1) + "s"; 
		time_msg = "time: " + time_spent_str + " :: " + time_remain_str; 
	        		
		$(this.trf_msg_cont).css('display', 'block');
		$(this.trf_msg).css('color', "#666");  	
		$(this.trf_msg).html(time_msg); 	
	}

	// Reset cursor 
	document.body.style.cursor = 'default';
};

// Adjust a progress bar 
Ctrl_Upl_Progr.prototype.display_upl_percentage = function() {
	width = this.present_upl_percentage + '%';
	$(this.id_bar).css('width', width); 
	console.log("\r\nPresent upload percentage = " + this.present_upl_percentage + "\r\n"); 
}; 

// use another specialized CtrlO for handling a special message display area on the web page 
Ctrl_Upl_Progr.prototype.display_msgs = function() {
	this.SO_Msg.show_msg(0, this.msg);
}; 

 
Explanations:
We just split the JSON response object into its components. The next article will show how we get these values on the server and send them back via Ajax. Components of the JSON response object are:

  • an indicator for potential errors detected by the PHP polling program
  • messages of different types (created during execution of the PHP polling program on the server)
  • progress information in form of a percentage value
  • a value for the transfer rate measured on the server
  • the total file size in bytes
  • the number of bytes received on the server so far
  • an end time value (date down to seconds) on the server when the file transfer was finished

We first transform both the uploaded bytes and the file size to KBytes or MBytes. The end time information is also used as a signal to stop the time loop for polling jobs on the client. This means that it will only be sent as a non blank string if our PHP polling program finds that the upload has finalized.

Please, note that we raise the number of allowed polling calls to the server when we are approaching the limit of allowed calls – if we still see any progress of the upload. Oops – what if the rate is so small that we fall below the discussed server’s limit of a 1% change during the polling time interval and progress information is not updated ? Then our monitoring would be stopped! Yes, this would be true if we did not adapt the polling period ….

Adaption of the polling time period

We just saw that it may become essential for our polling based progress monitoring to determine whether there is any progress at all. However, due to the parameter settings for the update interval of the progress information a too small polling period may lead to the false impression of zero progress over one or several polling intervals in case of slow connections.

This is a major reason for adapting the polling time interval to the transfer rate. We do this in our example after 3% of the file is loaded (assuming that our initial interval allows for such a progress during the first hundred polls. If it does not, than our polling interval really is much too small and we need to give it a higher initial value or increase the maximum initial number of allowed polls). In our simple adaption algorithm we set an ideal progress rate to 1.2% of the file size per polling interval. (To choose this a bit bigger than the limit of 1% for updates on the server is done on purpose!)

Furthermore, we only do something with our period if the polling time interval deviates more than 10% from the ideal time interval – and we never go below an interval of 1200 msec. In addition we limit changes to a maximum of 20% of the present value to avoid a jumpy behavior during the first steps of the transfer when the rate may change grossly. Thus, we try to change the polling time interval rather smoothly.

We leave it up to the kind reader to improve our simple adaption algorithm. I may say, however, that it works quite well for my aDSL and vDSL connections where reasonable rates are possible. I do not regard the lower limit of 1200 msec as a problem: If the whole file is loaded in one polling interval – hey, I am happy. Monitoring is needed for slower connections.

Note that we we deliver a parameter “0” to the callback of

setTimeout(this.get_status_by_ajax.bind(this), poll_soll, 0);

This kind of passing a parameter to a callback works in FF and Chrome. Unfortunately, it does not work in MS IE, Version <= 9. Fortunately, our method "get_status_by_ajax()" already takes care of an undefined function parameter.

Some Output

As indicated some received progress information is displayed in special DIV containers; the percentage information may be used to simulate a progress bar by a dynamical change of the width of some colored DIV.

Some other messages from the server are, however, handled by a special CtrlO for a message display and formatting area on our web page. The reader may design such an area by himself and write a related CtrlO. In our example we indicated the use of a message handling CtrlO by the statement “this.SO_Msg.show_msg(0, this.msg);”

Note that we did not discuss any handling of error messages from the server yet. We leave this to
the reader. You may use a boolean element “json_progr_result.err” of the JSON response object to indicate an error and its type – and react properly to it. We do not elaborate this here.

In this article we have shown some basic ingredients on the Javascript side for polling progress information about a file transfer from a PHP server. In the next article of this series

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – V

we shall have a look a the PHP side.

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – III

In the first 2 articles of this series I described a bit of what had to be done on the JS/jQuery client side to trigger the first phase (Phase I) of an Ajax controlled file upload (here: of a Zip container file). See:

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II

In this third article, we take a first glimpse at what has to be done on the server side – in our case in the PHP 5.4 target program of our Ajax request. There are of course very many and different ways to deal with the initial treatment of an uploaded file. What I normally do is to create a “File_Handler” object based on a class which encapsulates and provides all required methods. However, for the sake of a better understandability the code fragments presented below do not always follow a stringent OO line of code development and are sometimes very basic. We only show code elements that are of fundamental importance.

The reader may remember that the method of our Javascript CtrlO for controlling the upload form (see the last article) called a PHP target program named

handle_uploaded_init_files.php5?file

.
The attached GET-parameter distinguishes the first phase of the upload from several following phases. One of the initial things our PHP program should do is to open/access a PHP session, to start PHP output buffering and to check whether the $_GET element $_GET[‘file’] exists:

Excerpts from the PHP target program of Phase I – handle_uploaded_init_files.php5:

	$time_0 = microtime(true); 
	session_start();
	ob_start(); 
	....
	// Deal with phase I of the upload 
	if ( isset( $_GET['file'] )) {
		...
		$response = handle_transferred_file(); 
		ob_end_clean(); 
		echo $response;  
	}

(Side remark: Note that the attached Get parameter is nevertheless transferred via the POST mechanism of HTTP !)

$response represents a prepared encoded JSON-Object which is sent back to the JS-Ajax-client at the end of the program. By the “if”-statement we just distinguish the actions of Phase I from later phases where we deal with each file transferred in our large Zip-container file. The main work for our phase I is in our example obviously done inside a function “handle_transferred_file()”.

The use of “session_start()” seems to be quite reasonable. The PHP manual
http://php.net/manual/en/session.upload-progress.php
describes that upload progress information is supplied via $_SESSION. This is just the way the progress tracking of PHP 5.4 works!

However, we shall come back to this point at the bottom of this article and we shall see that things are not that simple. But, we may use the $_SESSION array also for keeping other interesting information. Anyway, using a PHP-session (=opening/accessing a session) will do no harm here.

But why do we need ob_start() ?
In the last article of our series the Ajax answer to the client was requested to have the form of a JSON object. So, our Ajax driven client program expects exactly one information stream in the form of a JSON object. However, if your own PHP code accidentally produces strings by executing some echo, print or print_r statements this rule will be violated. The client will receive the first PHP output as the expected Ajax JSON answer and
will not be able to parse it. This would result in an Ajax client error! Another source of unwanted and not JSON encoded output may come from the PHP engine itself which may create warnings and error messages during code execution. Therefore, it is a good habit to gather all such information in the output buffer and put it eventually as a special string element (as “system_mgs”) of an array which we shall encode as the final JSON object. See below.

A function and an array for producing the JSON answer of upload Phase I

Let us now turn to our function dealing with phase I. In our simple test case it basically does 2 things:

  • It creates a singleton object for performing progress tracking and later some file operations. This object then does the necessary things itself and produces an array which we in our example receive in the variable $response.
  • It enriches the response array with some information and encodes it as a JSON object.

That we have split the required actions into these steps is again more for the sake of illustration purposes.

function handle_transferred_file() {
	...
	// Some local variables 
	$ctrl_msg = '';
	
	// Includes
	require_once $my_include_path . 'class_file_handler.php5';
	require_once $my_include_path . 'class_file_handler_params.php5';
		
	// Parameter Singleton with parameters to control the file handling  	
	$F_Params = new FileHandlerParams(); 
		
	// File handler object (singleton) 	
	$F_Handler = new Basic_Class_File_Handler( $F_Params );
	if (isset($FCheck->msg)) {
		$ctrl_msg .= "\r\nFCheck initialized";
		$ctrl_msg .= "\r\nInitial msg = " . $FCheck->msg;
	}

	// Transfer the file to its directory location and unzip its contents 
	$F_Handler->check_and_save_uploaded_file(); 

	// Required time and response enrichment 		
	$time_f = microtime(true);
	$dtime_f = $time_f - $time_0;
	$F_Handler->ay_ajax_response_1["transfer_time"] = $dtime_f;
	
	// Response enrichment by system messages form the output buffer 
	$F_Handler->ay_ajax_response_1['sys_msg'] .= ob_get_contents(); 
	
	// return JSON encoded array 
	$response =json_encode($F_Handler->ay_ajax_response_1);
	return $response; 
}

This is fairly simple to understand: We load some class definitions – one for the handling of the file transferred into the $_FILE superglobal and one with a bunch of parameters. We then create a singleton object $F_Handler which does most of the required actions of Phase I. Its property “$F_Handler->ay_ajax_response_1” obviously is an array which is enriched by the contents of the PHP output buffer and some time information (just for illustration purposes). This array is eventually encoded as the required JSON answer object.

Typically the element $F_Handler->ay_ajax_response_1[‘sys_msg’] is used on the client side for tests only via the console.log() statement of Javascript.

Possible parameters of the File Handler object $F_Handler

What kind of parameters may be required? E.g.: the path of the directory where we want to save our transferred Zip-file and/or its contents on the server. The expected file ending. The maximum file size allowed. A parameter defining whether we really want to check the progress of the upload. The reader may think about more useful parameters.

In our example such parameters can be gathered and maintained via properties of the class “FileHandlerParams”. The F_Handler object may extract them from the respective Parameter object given as an argument to its constructor, and write them afterward into internal property variables.

Main elements of a Basic_File_Handler class

The “Basic_File_Handler_Class of our example may contain the following methods:

class Basic_Class_File_Handler {
	
	var $file_key; 		// Key of uploaded file in $_FILES[$key]
	var $file_name = ''; 	// Name of present file (uploaded or part of a pipeline)  
	var $file_mime = ''; 	// Mime-Type of present file 
	var $file_end = ''; 	// Suffix (ending) of present file 
	
	var $file_expected_end = array("csv", "zip");	// Allowed suffixes of the transferred file 
	
	var $file_size; 		// Size of present file 
	var $max_file_size = 100000000;	// Maximum allowed size 100MBytes
	var $check_progress = 0; 	// Check progress by means of PHP 5.4  ?
	var $sess_key_progress = ''; 	// The key of $_SESSION where to find upload progress infos 
	
	// Upload dir 
	var $upl_dir;  			// The given short name of the dir without root path 
	var $target_dir; 		// The full path of the target dir on the PHP server 

	var $ziel_file_name_oe; 	// Target name of file without ending and "."
	var $ziel_file_name; 		// Full target name of file 
	var $ziel_file_pfad; 		// Full target path relative to web domain dir  
	var $ziel_file_pfad_dom;	// Full target path rel. to PHP root on the server 

	var $root;   			// rel. path of present application to PHP application root
	var domain_root;		// rel. path of present application to Web domain root 
	
	var $upload_success = 0; 
	var $msg = ''; 
	var $sys_msg = ''; 
	var $err = 0; 
	var $err_msg = '';  
	
	// zip file info 
	var $num_extracted_files = 1; 
	var $file_pipeline = 0;  // will be set to 1 if the zip-file contains more than 1 files 
	
	// Ajax response array 
	var $ay_ajax_response_1 = array();   // For phase 1 	
	
	.....
			
// Main methods 
// -----------
	// constructor 
	function __construct($Params)  {
 		...
	}

	// function to check properties of the uploaded file and save it in a target directory on the server  
	function check_and_save_uploaded_file() {
		....
	}		
	
	// Method to prepare Ajax (JSON) response array for phase 1 of upload 
	function prepare_ajax_response_phase_1() {
		...
	}
		
	// Check existence of transferred file in $_FILES
	function check_file_existence_and_props() {
		...
	}
	
	// Check suffix = file ending 
	function get_and_check_file_suffix() {
		...
	}
	
	// Check File_Size
	function check_file_size() {
		...
	}
		
	// Delete all existing files in the upload dir 
	function delete_files_from_upload_dir() {
		...
	}
	
	// Move the uploaded file or the content files of an uploaded zip file to the target directory 
	function move_file_to_upload_dir() {
		...
	}
	
	// function to determine the number of extracted files and the name of the next file to handle  
	function determine_num_files_and_next_file_name() {
		...
	}
	
	// Method to handle a zip archive 
	function handle_zip_archive( $dest_file, $dest_dir) {
		...
	}
	...
	...

// End of class definition  
}

We shall look at some details of these methods in the articles to come.

The key to session information about the progress of the ongoing upload progress

The constructor of our Basic_File_Handler class gets the task to read in parameters. In addition we use it here and try to retrieve some (initial?) progress information from the $_SESSION array – just for learning purposes. Actually, we shall see that this trial will NOT give us any progress information at all or only a trivial one.

function __construct($Params_ext) {
	
	// read external parameters 
	$num_args = func_num_args(); 
	if (num_args == 1 ) {
		$Params = func_get_arg(0);
		if ( is_object($Params) && get_class($Params) == "FileHandlerParams" ) {
			$this->file_key  = $Params->file_key; 
			if( is_array($Params->file_types) && count($Params->file_type) > 0 ) {
				$this->file_type = $Params->file_types; 
			}
			$this->upl_dir 	= $Params->upload_dir . "/";
			$this->max_file_size	= $Params->max_file_size; 
			$this->check_progress= $Params->check_progress; 
		}
	}
	// Error treatment 
	else {
		...
		$this->ay_ajax_response_1[sys_msg] .= "\r\nWrong Parameter object!"; 
	}		
		
	// Test the progress information in $_SESSION (Is this reasonable here ???) 
	if ( $this->check_progress == 1 ) {
		// Get the required string end for the key of progress info in the $_SESSION array
		// This information was delivered via $_POST by the Ajax client program 
		$this->sess_key_progress = '';
		$key_POST = ini_get("session.upload_progress.name");
		if (isset( $_POST[$key_POST] ) ) {
			$this->sess_key_progress .= ini_get("session.upload_progress.prefix"). $_POST[$key_POST];
			$this->sys_msg .= "<br>sess_key_progress = " . $this->sess_key_progress;  
			$this->ay_ajax_response_1['sess_key_progress'] = $this->sess_key_progress;  
			$_SESSION['progress_key'] = $this->sess_key_progress;
			
			// Write a test value into a special Session variable 
			$current = -1;
			if (isset($_SESSION[$this->sess_key_progress]) && !empty($_SESSION[$this->sess_key_progress])) {
				$current 	= $_SESSION[$this->sess_key_progress]["bytes_processed"];
				$total 		= $_SESSION[$this->sess_key_progress]["content_length"];
				$current	= ($current < $total) ? ceil($current / $total * 100) : 100;
			}
			$this->sys_msg .= "<br>Initial progress value of the file transfer was " . $current; 
		}
	}	
		
	// set the TARGET DIR for saving the file
	$this->target_dir 	= $this->root . $this->upl_dir;
			
	return;
}

 
$this->check_progress is set by the parameters transferred.

Note that progress tracking and the delivery of related information in the $_SESSION array requires the following setting in the php.ini file (“/etc/php5/apache2/php.ini” on Opensuse)

; Enable upload progress tracking in $_SESSION
; http://php.net/session.upload-progress.enabled
session.upload_progress.enabled = On

In the lower part of the PHP code we see how the key for the element of the $_SESSION array which contains information about the upload progress is composed:

We first need a parameter from the php.ini file which we read by ini_get(“session.upload_progress.prefix”). Then we need another parameter of our php.ini file on the server which gives us a special index of the $_POST array:
$key_POST = ini_get(“session.upload_progress.name”).
This index points to an element of the $_POST array which (hopefully) contains a unique identifier of our presently progressing upload process. We must submit this identifier by our Ajax client method initiating the whole file transfer; see the last article in this series. Eventually, we combine both pieces of information to get the index ($this->sess_key_progress) of an element of the $_SESSION array which will contain progress information about our upload.

In the code above we write this information into our output array as part of an Ajax response.

When and how is the progress information
available in the $_SESSION array?

The next code section then uses our composed key to read a test status value of the upload (at the point of code execution) from the $_SESSION array. Now let us assume that we really used our coding above – what values would we get ? Some integer numbers between 0 and 100 ?

The answer clearly is NO – we would always get “-1”. Why ?

You may say – maybe the connection is too fast and the $_SESSION variables may get erased when the upload has finalized. Well there is some truth in this also:

If we really wanted to test progress tracking in a LAN we may need to reduce the bandwidth of your network connection. Information about how one can achieve this can be found in the article
Geschwindigkeit/Übertragungsrate eines Netzwerkinterfaces unter Linux reduzieren

And yes – depending on the parameter setting in the “php.ini”-file the progress related elements of the $_SESSION array may get eliminated:

; Cleanup the progress information as soon as all POST data has been read
; (i.e. upload completed).
; Default Value: On
; Development Value: On
; Production Value: On
; http://php.net/session.upload-progress.cleanup
session.upload_progress.cleanup = On

But even if you reduced your bandwidth considerably in comparison to your transferred file you would nevertheless only get “-1” ! And even if you set the cleanup parameter to “Off” you would only get a trivial answer: 100.

The real reason for our failure is simply that the code execution of our PHP target program only starts when all data sent via the POST mechanism are completely received – this includes the file data!

This is very logical! The core purpose of the PHP job triggered by our file upload form in the last article is to deal with the uploaded file. It cannot be used for progress tracking by principle! We need a different and independent mechanism – in our case independent polling jobs.

So, when the PHP code listed above is executed on the server the file is already fully uploaded and the $_SESSION elements of $_SESSION which contained the progress information may no longer exist if the cleanup parameter is ON in php.ini settings!

The fact that the PHP job started by the upload form is of no use for progress tracking has two important implications:

First: By what should or could the PHP session for progress tracking be started at all in our Ajax controlled job environment? One assumption could be that the PHP engine does it by itself as soon as it somehow recognizes upload circumstances. However, this is not the case – and uncontrolled automatic session starts would actually introduce security risks into PHP. In fact we have reached an important point :

The PHP session must have been started already before our Ajax controlled file transmission from the client starts!

Otherwise the whole progress tracking process will not work! It uses a PHP session that must have been opened before!

How could we do this? Now, I may remind the kind reader about a note in the last article: Our HTML page with the upload form was created by a PHP program – which itself may use method of the classes of a template engine like Smarty. So, the PHP program that creates our initial web page already could open the required PHP session! An alternative would be that our client starts a precursor Ajax job with the sole purpose of opening a PHP session. Not very efficient, but possible.

And another aspect has become very clear now that would also be valid for independent polling jobs:

If we want to access the progress information in the $_SESSION array about the ongoing upload process we MUST have the $_
POST information about the upload process identifier available as the first information reaching the server – i.e. before the file data themselves start running in! This is the very reason why we had to take care about the order by which data are sent from the client to the server – see the related remarks in the previous article!

Note in addition that we did not destroy the session or unset its cookie at the end of our target program program function for Phase I. The reason is that we may use the session also in later phases.

Enough for today. Please, note also that the only substantial and effective things we have discussed so far on the PHP side were:

  • to access a hopefully already existing (!) PHP session,
  • to initiate the PHP output buffer

However, we have learned something about the key we use for accessing progress information in the $_SESSION array and understood its relation to a $_POST parameter, which must be provided by the client.

We shall come back to the treatment of the fully uploaded file by the methods listed above in a later article.
In the next article

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – IV

we shall instead look at the required sequence of Ajax controlled “polling” jobs started by the client to retrieve information about the upload progress status from the server. Independently of our main PHP job ….