PHP and web application security – bad statistics and wrong conclusions

Sometimes I have discussions with developers of a company working mainly with Java. As I myself sometimes do development work with PHP I am regarded more or less as a freak in this community. Typical arguments evolve along the lines:

"PHP does not enforce well structured OO code, no 4-layer-architecture built in, problems with scalability", etc.... I do not take these points too seriously. I have seen very badly structured Java OO code, one can with proper techniques implement web services in a kind of logical 3rd layer on special servers and Facebook proves PHP scalability (with some effort), etc...

What is much more interesting for me these days is the question how security aspects fit into the use of different programming languages. And here comes the bad news - at least according to statistics published recently by the online magazine Hacker News - see
http://thehackernews.com/2015/12/programming-language-security.html

The statistics on OWASP 10 vulnerability types of the investigated PHP code looks extremely bad there - compared to the investigated Java code examples. I admit that this is an interesting result for hackers and that it is somewhat depressing for security aware PHP developers.

However, is this the bad statistics the fault of the programming language?

I doubt it - despite the fact that the named article recommends to "Choose Your Scripting Language Wisely". I would rather recommend: Educate your PHP developers properly and regularly, implement a proper quality assurance with special security check steps based on vulnerability scanning tools and invest in regular code reviews. Why?

The investigation revealed especially large deviations in the fields of XSS, SQL-injection, command injection (major elements on the OWASP 10 list). The countermeasures against the named attack vectors for PHP are all described in literature and very well known (see e.g. the books "PHP Sicherheit" of C.Kunz, S.Esser or "Pro PHP Security" of Snyder, Myer, Southwell). One of the primary key elements of securing PHP applications against attacks of the named types is a thorough inspection, analysis and correction treatment of submitted GET/POST-parameters (and avoidance of string parameters wherever possible). Never trust any input and escape all output! Define exceptions wisely and rewrite sensitive string elements according to your rules. Check whether input really comes from the right origin - e.g. your own domain, etc., etc.

Whether all relevant security measures are implemented in the PHP code of a web application has therefore more to do with the mentality, technical ability, the knowledge and on the negative side with the laziness of the programmer than with the programming language itself. As at least the technical capability is a matter of education, I conclude:

Tests regarding type, value range, length and of course tests of the contents of received string variables and e.g. image source references plus sanitizing/elimination/deactivation of problematic contents as well as the proper use of respective available library functions for such tests should be part of regular PHP training programs. In addition the use of web application scanning tools like OWASP's ZAP scanner or the Burp Suite Pro (if you have money to afford the latter) should be trained. Such tools should become part of the QA chain. As well as educated penetration testers with the perspective of the attacker .... The money a SW-company invests for such educational measures is well invested money.
See for the significant impact of education e.g.:
https://seclab.stanford.edu/websec/scannerPaper.pdf

I would regard the statistical results discussed in the "Hacker News" article much more conclusive if we were provided additional information about the type of applications analyzed and also the size and type of the companies behind the application development. And whether and what type of QA efforts were used. This would give a much better indication of why the Java code showed more quality regarding the prevention of OWASP 10 attacks. One reason could e.g. be that Java applications very often are developed for enterprise purposes - and bigger companies typically invest more time and effort into QA ...

So, another valid interpretation of the presented statistics could be that the QA for typical PHP web application SW is on average worse than for Java SW. I admit that such a finding would also be very interesting - but it does not prove that one cannot write secure Web applications with PHP or that the production of secure code is for whatever reasons easier with Java.

In addition the presented number of bugs per MB itself is questionable: if you only look at 3 bad PHP examples and 1 good Java example you may get the same type statistics - but it would be totally meaningless. The distribution of PHP-, Java-, JS-code etc. among the statistical sample is, however, nowhere discussed in the named article - neither in number of applications nor in MB percentages.

Therefore: Without further information the implied conclusion that already the proper choice of a web scripting language would help to improve security of web applications appears is misleading.

To improve the mood of PHP developers having read the article in "Hacker News": Have a look at the results of a similar investigation presented at this link
http://info.whitehatsec.com/rs/whitehatsecurity/images/statsreport2014-20140410.pdf

See also:
https://blog.whitehatsec.com/a-security-experts-thoughts-on-whitehat-securitys-2014-web-stats-report/
https://www.scriptrock.com/blog/which-web-programming-language-is-the-most-secure
https://threatpost.com/security-begins-with-choice-of-programming-language/105441/

It may help, really !

[But keep in mind: Only trust statistics you have manipulated yourself.]

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – VI

In the previous articles of this series about an Ajax controlled file upload with PHP progress tracking

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – V
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – IV
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – III
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I

we have shown how we can measure and display the progress of a file upload process with a series of Ajax controlled polling jobs and the progress tracking features of PHP > 5.4. At least in our test example this worked perfectly.

However, for practical purposes and especially when our server users deal with large files we must in addition take better care of some limiting PHP parameters on the server. Both a good server admin and a program developer would, of course, try to find out what file sizes are to be expected on a regular basis and adjust the server parameters accordingly. However, you never know what our beloved users may do. What happens if we talked about file sizes of less than 100MB and suddenly a file with 200MB is transferred to the server?

For which limiting PHP parameters on the server may we run into serious trouble?

Due to security considerations the PHP module interaction with incoming data streams is limited by parameters set for the Apache server. The relevant configuration file is e.g. on Opensuse located at

/etc/php5/apache2/php.ini

The most important limits (set in different sections of the file) are:

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate unexpectedly
; long running scripts.
; Default Value: -1 (Unlimited)
; Production Value: 60 (60 seconds)
; http://php.net/max-input-time
max_input_time = 200
 
; Maximum size of POST data that PHP will accept.
; Its value may be 0 to disable the limit. It is ignored if POST data reading
; is disabled through enable_post_data_reading.
; http://php.net/post-max-size
post_max_size = 200M
 
; Maximum allowed size for uploaded files.
; http://php.net/upload-max-filesize
upload_max_filesize =150M
 
; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
memory_limit = 500M

The description of the first parameter above is somewhat unclear. What is meant by the "time spent on parsing request data"? Is this a part of the (also limited) execution time of the PHP target program of our Ajax transaction? Or is this limit imposed on the time required to read incoming POST data and to fill the $_POST array? If the latter were true a small bandwidth could lead to a violation of the "max_input_time" limit ...

Regarding the second parameter the question turns up, whether this limit is imposed on all transferred POST data including the file data?

The third parameter seems to speak for itself. There is a limit for the size of a file that can be transmitted to the server. However, it is not clear how this parameter affects real world scenarios. Does it stop a transfer already before it starts or only when the limit is reached during the transfer?

Regarding the 4th parameter we may suspect that it becomes important already during the handling (reading, parsing) of the incoming POST data. So, how much of memory (RAM) do we need at the server to handle large files during an upload process?

Warning regarding PHP parameter changes for multi-user situation on real world servers

We were and are discussing a privileged situation in this article series: Only one user uploads exactly one big Zip-container file to a server.

In such a situation it is relatively safe to fiddle around with PHP parameters of the central "php.ini" file (or PHP parameter settings in directory specific files; see the last section of this article). However, as an administrator of a server you should always be aware of the consequences of PHP parameter changes, e.g for memory limits, in a multi-user environment.

In addition you must also take into account that our code examples may be extended towards the case that one user may upload multiple files in parallel in one Ajax transaction.

Remarks on "max_input_time" - you can probably ignore it!

If you look up information about "max_input_time" available on the Internet you may experience that some confusion over the implications of this parameter remains. Especially as PHP's own documentation is a bit contradictory - just compare what is said in the following manual pages:

 
Therefore, I tested a bit with files up to 1 GByte over slow and fast connections to PHP servers on the Internet. I came to the conclusion that the answer in the following "stackoverflow" discussion
http://stackoverflow.com/questions/11387113/php-file-upload-affected-or-not-by-max-input-time
describes the server behavior correctly. This means:

This parameter has no consequences with respect to connection bandwidth and the resulting upload time required for file data: It does not limit the required upload time. Neither does it reduce the amount of allowed maximum execution time of the PHP program triggered at the end of the file transfer process to the server.

"max_input_time" imposes a limit on the time to read/parse the data

  • after they have completely arrived at the server
  • and before the PHP program, which shall work with the data, is started.

This "parsing" time normally is very small and the standard value of 60 secs should be enough under most circumstances. If these findings are true we do not need to care much about this parameter during our file transfer process to the server. A value of 60 secs should work even for large files of 1 GB ore more on modern servers. At least for a server with sufficient resources under average load.

See also:

 
However, I can imagine circumstances on a server with many users under heavy load, for which this parameter nevertheless needs to be adjusted.

What does the PHP documentation say about the parameters "post_max_size", "upload_max_filesize" and "memory_limit"?

Regarding these parameters we get at least some clear - though disputable - recommendation from the PHP documentation. At

 
we find the following explanation for "post_max_size":

Sets max size of post data allowed. This setting also affects file upload. To upload large files, this value must be larger than upload_max_filesize. If memory limit is enabled by your configure script, memory_limit also affects file uploading. Generally speaking, memory_limit should be larger than post_max_size. When an integer is used, the value is measured in bytes. Shorthand notation, as described in this FAQ, may also be used. If the size of post data is greater than post_max_size, the $_POST and $_FILES superglobals are empty. This can be tracked in various ways, e.g. by passing the $_GET variable to the script processing the data, i.e. <form action="edit.php?processed=1">, and then checking if $_GET['processed'] is set.

Off topic: For those who find the track-recommendation in the last sentence confusing as it refers to $_GET, see e.g.

 
You can add parameters to your URL and these parameters will appear in $_GET, but if you decided to use the POST mechanism for data transfer these URL-parameters are included in the POST data mechanism of HTTP.

The recommendation for memory sizing is misleading in case of file uploads!

Following the recommendation quoted above would lead to the following relation for the PHP setup:

memory_limit > post_max_size > ( upload_max_file_size * number of files uploaded in parallel ).

Regarding the right side: My understanding is that "upload_max_file_size" sets a limit for each individual file during an upload process. See

 
Actually, I find the recommendation for the parameter "memory_limit" very strange. This would mean that somebody who has to deal with an upload file with a size of 2 GByte would have to allow for memory allocation for a single PHP process in the RAM > 2 GByte. Shall we take such a requirement seriously?

My answer is NO ! But, of course, you should always test yourself ....

To me only the last relation on the right side of the relation chain makes sense during an upload process. Of course PHP needs some RAM and during file uploads also buffering requires sufficient server RAM. But several GByte to control a continuous incoming stream of data which shall be saved as a file into a directory (for temporary files) on the server? No way! I did some tests - e.g. limit the memory to 32 MB and successfully upload a 1 GB file. Therefore, I agree completely with the findings in the following article:

 
See also:

 
So:

Despite you need RAM for buffering during file uploads it is NOT required to use as much physical RAM as the size of the file you want to upload.

However, it may be wise to have as much RAM as possible if you intend to operate on the file as a whole. This may e.g. become important during phases when a PHP program wants to rewrite file data or read them as fast as possible for whatever purpose. A typical example where you may need sufficient memory is image manipulation.

Nevertheless: Regarding the file transfer process to the server itself the quoted recommendation is in my opinion really misleading. And: Do not forget that a high value for "memory_limit" may lead to server problems in a multi-user situation.

"post_max_size" and "upload_max_filesize" as the main limiting PHP parameters for file uploads

So, only the following condition remains:

post_max_size > upload_max_file_size * number of files uploaded in parallel

But this condition should be taken seriously! There are several things that need to be said about these parameters.

  1. A quick test shows: "post_max_size" imposes a limit on all POST data transferred from client - including file data.
  2. Even for situations in which only one file is uploaded I personally would choose "post_max_size" to be several MBs bigger than "upload_max_filesize". Just to account for overhead.
  3. In case of an upload of multiple files in parallel (i.e. a situation, which we have not studied in this article series) you have to get an idea about the typical size and number of files to be uploaded in parallel. In such a situation you may also want to adjust the parameter

    ; Maximum number of files that can be uploaded via a single request
    max_file_uploads = 20

  4. There may be differences depending on the PHP version of how and when the server reacts to a violation of either of both parameters. For PHP 5.4 it seems that the server does not allow for an upload if either of the parameters is violated by the size of the transferred file(s) - meaning: the upload does not even start. This in turn may lead to different error situations on the server and messages issued by the server - depending on which parameter was violated.
  5. From a developer's perspective it is a bit annoying that the PHP servers reaction to a violation of "upload_max_filesize" is indeed very different from its reaction a violation of "post_max_size". See below.

Server reactions to violations of "post_max_size" and "upload_max_filesize"

We need to discuss a bit the reactions of a PHP server towards a violation of the named parameters before we can decide how to react within our PHP or Javascript programs in the course of an Ajax transaction.

Server reaction to a violation of "upload_max_filesize"
The Apache/PHP server reacts to a violation of "upload_max_filesize" by a clear message in

$_FILES['userfile']["error"]

where 'userfile' corresponds to the "name" attribute of the HTML file input element. A reasonable way how to react to PHP error messages in $_FILES by PHP applications is described in the highest ranked comment of
http://php.net/manual/en/features.file-upload.errors.php
and also here
https://blog.hqcodeshop.fi/archives/185-PHP-large-file-uploads.html

Server reaction to a violation of "post_max_size"
What about a violation of "post_max_size"? We can only react reliably to an error via our PHP target programs if an error number or a clear, structured message is provided. Unfortunately, this is not the case when the sum of uploaded data via POST becomes bigger than "post_max_size". When the server detects the violation no content at all is made available in $_POST or $_FILES. So, we have no error-message there a PHP program could react to.

However, we can combine

  • a test for emptiness of the superglobals $POST and $_FILES
  • with some HTTP information from the client, which is saved in $_SERVER,

to react properly in our PHP programs. Such a reaction within our Ajax transactions would naturally include

  • the creation of an error code and an error-message
  • and sending both back within the JSON response to the Javascript client for error control.

When we make a POST request to the server a value of the POST content size is provided by the client and available via the variable

$_SERVER['CONTENT_LENGTH'].

See:

 
So, for the purpose of error control we will need to add some test code to the "initial" PHP target program "handle_uploaded_init_files.php5" of our Ajax transaction which started the file upload.

Reasonable reactions of our PHP upload and polling programs to a violation of "post_max_size"

Remember that our initial Ajax transaction for upload triggered the server file "handle_uploaded_init_files.php5". Therefore, we should some additional code that investigates the violation of post_max_size" there. This would probably look similar to:

if (
	isset( $_SERVER['REQUEST_METHOD'] )      &&
        ($_SERVER['REQUEST_METHOD'] === 'POST' ) &&
        isset( $_SERVER['CONTENT_LENGTH'] )      &&
        ( empty( $_POST ) )
 ) {
	$max_post_size = ini_get('post_max_size');
	$content_length = $_SERVER['CONTENT_LENGTH'] / 1024 / 1024;
	if ($content_length > $max_post_size ) {
		....
		// Our error treatment ....
		$err_code = ....;
		// create an error message and send it to the Ajax client 
		$err_post_size_msg = ".....";
		....
	}

	....
	....
	// transfer the error code and error message to some named element of the JSON object 
	....
	$ajax_response['err_code'] = $err_code;
	$ajax_response['err_msg'] = $err_post_size_msg;
	.....
	$response = json_encode($ajax_response);
	echo $response;
	exit;
}

 
See also:

&nbsP;
Note that we cannot assume a certain timing of the reaction of the main program in comparison to our polling jobs. It may happen that we have already started the polling sequence before the error messages from our first Ajax transaction arrive at the client. Therefore, also our polling jobs "check_progress.php5" should be able to react to empty superglobals $_POST and $_FILES :

if ( ( empty( $_POST ) ) && empty ( $_FILES ) ) {
	// Our error treatment ....
	// create an error message and send it to the Ajax client 
	// refer to messages that may turn up in parallel from the main PHP program
	....
}

 
The different Javascript client methods which receive their respective Ajaj messages should evaluate the error messages and error numbers from the server, display them and, of course, stop the polling loop in case it is still active. As these are trivial programming steps we do not look deeper into them.

Avoid trouble with limiting PHP parameters before starting the file upload

Although we can react to error situations as described above I think it is better to avoid them. Therefore, I suggest to check file size limits before starting any upload process.

In our special situation with just one big Zip-file to upload we can initiate a file size limit check on the server as soon as we choose the file on the client. This means that the Javascript client must be enabled to react to the file selection action and request some information about the parameters "post_max_size" and "upload_max_filesize" from the server. In addition we need a method to compare the server limits with the size of the chosen file.

Looking into
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress control – II
we see that we had defined a proper Javascript Control Object [CtrlO] for the upload form

<form id="form_upload" name="init_file_form"  action="handle_uploaded_init_files.php5" method="POST" enctype="multipart/form-data" >

 
which - among other things - contains the file selection input tag:

<input type="file" name="init_file" id="inp_upl_file" >

 
However, we had not assigned any method to the file selection process itself. We are changing this now:

function Ctrl_File_Upl(my_name) {
	
	this.obj_name = "Obj_" + my_name; 
	
	// Controls related to GOC and dispatched object addresses
	this.GOC = GOC;
        this.SO_Tbl_Info = null; 
        this.SO_Msg      = null; 
			
	// ay to keep the selected file handles 
	this.ay_files = new Array(); 
		
	// msg for 1st Ajax phasefor file upload; 
	this.msg1 = ''; 
		
	// Timeout for file transfer process
	// this.timeout = 500000; // internet servers  
	this.timeout = 300000; 
		
	// define selectors (form, divs) 
	this.div_upload_cont_sel 	= "#" + "div_upload_cont";
	this.div_upload_sel 		= "#" + "div_upload";
	this.p_header_upload_sel 	= "#" + "upl_header" + " > span";
			
	this.form_upload_sel 		= "#" + "form_upload";
	this.input_file_sel 		= "#" + "inp_upl_file";
	this.upl_submit_but 		= "#" + "but_submit_upl";

	this.hinp_upl_tbl_num_sel	= "#" + "hinp_upl_tbl_num";			
	this.hinp_upl_tbl_name_sel	= "#" + "hinp_upl_tbl_name";			
	this.hinp_upl_tbl_snr_sel	= "#" + "hinp_upl_tbl_snr";			
	this.hinp_upl_succ_sel 		= "#" + "hinp_upl_succ";			
	this.hinp_upl_run_type_sel 	= "#" + "hinp_upl_run_type";			
	this.hinp_upl_file_name_sel 	= "#" + "hinp_upl_file_name";			
	this.hinp_upl_file_pipe_sel 	= "#" + "hinp_upl_file_pipe";

	// display the number of extracted and processed files 
	this.num_open_files_sel		= '#' + "num_open_files";
	this.num_extracted_files_sel 	= '#' + "num_extracted_files";
	
	// Other objects on the web page - progress area 
	this.trf_msg_cont	= '#' + "trf_msg_cont";
	this.trf_msg		= '#' + "trf_msg";
	this.imp_msg_cont	= '#' + "imp_msg_cont";
	this.imp_msg		= '#' + "imp_msg";
				
	// Status (!) message box (not the right msg box) 
	this.status_div_cont = '#' + "status_div_cont";
	this.id_progr_msg_p  = "#progr_msg"; 
	
	//progress bar 
	this.id_bar = "#bar"; 
			
	// right msg block
	this.span_main_msg	= "span_msg"; 
	
	// variables to control the obligatory check of the file size 
	this.file_size_is_ok = 1; 
			
	// variables for the Ajax response 
	this.upl_file_succ 	= 0; 
	this.upl_file_name 	= ''; 
        
	// File associated variables
	this.file_name      = '';
        this.file_size_js   = 0; 	// file size detected by JS
        this.file_size      = '';	// file size detected by server
        this.allowed_file_size = 0;	// allowed file size for uploads on the server  
        
	// Processing of files     
        this.num_extracted_files = 0;
	this.file_pipeline 	 = 0; 
	this.import_time 	 = 0; 
	this.transfer_time 	 = 0; 
        
	this.name_succ_dir = ''; 
	this.num_open_files = 1;  

	// transfer time measurement 
	this.date_start = 0;   
	this.date_end = 0; 
	this.ajax_transfer_start = 0; 
	this.ajax_transfer_end 	 = 0; 
			
	// database import time measurement 
	this.date_data_import_start = 0; 
	this.date_data_import_end = 0; 
	this.data_import_start = 0; 
	this.data_import_end = 0; 
			
	this.transfer_time	= 0; 
	this.processing_time = 0; 
	this.time_start = 0; 
			
	// Determine URL for the Form 
	this.url = $(this.form_upload_sel).attr('action'); 
	console.log("Form_Upload_file - url = " + this.url);  				

	// Register methods for event handling 
	this.register_form_events(); 
			 
}	
	
// Method to start uploading the file  
// -------------------------------------------------------------------
Ctrl_File_Upl.prototype.register_form_events = function() {
			
	$(this.input_file_sel).click(
		$.proxy(this, 'select_file') 
	);
			
	$(this.input_file_sel).change(
		$.proxy(this, 'fetch_allowed_file_size') 
	);
	
	$(this.upl_submit_but).click(
		$.proxy(this, 'submit_form') 
	);
				
	$(this.form_upload_sel).submit( 
		$.proxy(this, 'upl_file') 
	); 
					
}; 

 
The reader recognizes that in contrast to the version of the CtrlO "Ctrl_File_Upl" discussed in previous articles of this series we have added some selector IDs for fields of some other web page areas. But the really important change is an extension of the methods for additional events in "Ctrl_File_Upl.prototype.register_form_events()":

First, we react to a click of the file selection button of the file input field. This only serves the purpose of resetting fields and message areas on the web page. But we react also to the file selection itself by using the change event of the file input field. This triggers a method "fetch_allowed_file_size()" which retrieves the parameter "upload_max_filesize" from the server.

Note:
We assume here that the server admin was clever enough to set post_max_size > upload_max_filesize!
Therefore, we only will perform a file size comparison with the value of "upload_max_filesize". If you do not trust your server admin just extend the methods and programs presented below by an additional and separate check for file sizes bigger than "post_max_size". This should be an easy exercise for you.

Now, let us have a look at the new methods of our Javascript CtrlO :

	
// Method to react to a click on the file selection box 
// ---------------------------------------------------- 
Ctrl_File_Upl.prototype.select_file = function (e) {
	// Call method to reset information and message fields 
	// Note: The following method also deactivates the file submit button !  
	this.reset_upl_info(); 
};

	
// Method to check whether file size is too big   
// ---------------------------------------------
// We check whether the file size is too big 
Ctrl_File_Upl.prototype.fetch_allowed_file_size = function (e) {
			
	this.file_size_is_ok = 0; 
			
	// size of the file im MByte determined on the client 
	this.file_size_js = $(this.input_file_sel)[0].files[0].size/1024/1024;
	console.log("actual file size of chosen file = " + this.file_size_js); 	
			
	// Now trigger an Ajaj transaction 
	var ajax_url = "../func/get_allowed_file_size.php5"; 
	var form_data = ''; 
			
	// 03.07.2015: we avoid setup as this would be taken as the standard for subsequent Ajax jobs 
	$.ajax({
                //contentType: "application/x-www-form-urlencoded; charset=ISO-8859-1",
                // context:  Ctrl_Status
                url: ajax_url, 
		context:  this, 
		data: form_data, 
		type: 'POST', 
		dataType: 'json', 
                success: this.response_allowed_file_size, 
                error: this.error_allowed_file_size
        });
};

// Method for Ajaj error handling during file size check transaction 		
// --------------------------------------------------------------	
Ctrl_File_Upl.prototype.error_allowed_file_size = function(jqxhr, error_type) {
			
	// Reset the cursor 
	$('body').css('cursor', 'default' ); 

	// Error handling
	console.log("From Ctrl_File_Upl ::  got Ajax error for fetch_allowed_file_size" );  
	var status = jqxhr.status; 
	var status_txt = jqxhr.statusText; 
	console.log("From Ctrl_File_Upl.prototype.error_allowed_file_size() ::  status = " + status );  
	console.log("From Ctrl_File_Upl.prototype.allowed_file_size ::  status_text = " + status_txt ); 
	console.log("From Ctrl_File_Upl.prototype.allowed_file_size ::  error_type = " + error_type ); 
			
	var msg = "<br>Status: " + status + "  Status text: " + status_txt;    
	this.SO_Msg.show_msg(1, msg); 
};

// Method for Ajaj rsponse handling after file size check transaction 		
// --------------------------------------------------------------			
Ctrl_File_Upl.prototype.response_allowed_file_size = function (json_response, success_code, jqxhr) {
	
	// Reset the cursor 
	$('body').css('cursor', 'default' ); 
				
	var new_msg; 
	var status = jqxhr.status; 
	var status_txt = jqxhr.statusText; 
	console.log("response_allowed_fsize: status = " + status + " , status_text = " + status_txt ); 	

	// The allowed file size on the server
	this.allowed_file_size = json_response['allowed_size'];	
	// parseInt required due to possible MB or GB endings on the server 
	this.allowed_file_size = parseInt(this.allowed_file_size); 
	console.log("allowed file size on server = " + this.allowed_file_size); 	

	// size comparison
	// ----------------
	if ( this.file_size_js > this.allowed_file_size ) {
		this.file_size_is_ok = 0; 
		new_msg = $(this.span_main_msg).html();
		if (new_msg == undefined) {
			new_msg = ""; 
		}
		new_msg += "<br><span style=\"color:#A90000;\">File size too big.</span><br>" +
		"The server allows for files with a size ≤ " +  	
		parseFloat(this.allowed_file_size).toFixed(2) + " MB." + "<br>" + 
		"The size of the chosen file is " + parseFloat(this.file_size_js).toFixed(2) + " MB." + "<br><br>" + 
		"<span style=\"color:#A90000;\">Please choose a different file or reduce the contents !</span>" + "<br><br>" + 
		"If you permanently need a bigger file size limit on the server, please contact your administrator"; 
					
		this.SO_Msg.show_msg(0, new_msg); 
				
	// file size within limits 
	// -------------------------
	else {
		this.file_size_is_ok = 1; 
					
		new_msg = $(this.span_main_msg).html();
		if (new_msg == undefined) {
			new_msg = ""; 
		}
		new_msg += "<br><span style=\"color:#007700;\">File size within server limits.</span><br>" +
		"The server allows for files with a size ≤ " +  parseFloat(this.allowed_file_size).toFixed(2) + " MB." + "<br>" + 
		"The size of the chosen file is " + parseFloat(this.file_size_js).toFixed(2) + " MB." + "<br><br>" + 
		"<span style=\"color:#007700;\">Use the "Start Upload" button to start the file upload!</span>"; 
					
		this.SO_Msg.show_msg(0, new_msg); 
					
		// reactivate the submit button 
		// -----------------------------
		$(this.upl_submit_but).on("click", $.proxy(this, 'submit_form') ); 
		$(this.upl_submit_but).css("color", "#990000"); 
	}
};
	
// Method to reset some form and information fields on the web page 
// ---------------------------------------------------------------
// We have to reset some form and message fields
Ctrl_File_Upl.prototype.reset_upl_info = function() {
				
	var msg_progr = ''; 
	$(this.id_progr_msg_p).html('');
				
	var msg_trf = ''; 
	$(this.trf_msg_cont).css('display', 'none');
	$(this.trf_msg).css('color: #666'); 
	$(this.trf_msg).html(msg_trf);
		    	
	var msg_imp = ''; 
	$(this.imp_msg_cont).css('display', 'none'); 
	$(this.imp_msg).css('color: #666'); 
	$(this.imp_msg).html(msg_imp); 
		    	
	// Deactivate the "Start Upload" Button 
	// ------------------------------------
	$(this.upl_submit_but).off("click"); 
	$(this.upl_submit_but).css("color", "#BBB"); 

	// Reset also the main message area  
	// ----------------------------------------------
	this.SO_Msg.show_msg(0, ''); 

};	

 
This is all pretty straightforward and parts of it are already well known of our previous descriptions for handling the Ajaj interactions with the server by the help of jQuery functionality.

A short description of what happens is:

  • When you click on the button of the file selection input field contents of fields in the message area of our web page and information fields about upload progress are reset as we assume that a new upload will be started.
  • During reset also the form's submit button to start a file upload via Ajax/Ajaj is disabled. Note that we use jQuery's "off('event')"-functionality to to this.
  • As soon as the user selects a specific file we trigger a method which determines and saves the size of the chosen file to a variable and starts an Ajax transaction afterwards. This Ajax interaction calls a target PHP program "get_allowed_file_size.php5" in some directory.
  • The JSON-response of the PHP program is handled by the method
    Ctrl_File_Upl.prototype.response_allowed_file_size.

      The main purpose of this method is to make a comparison of the already determined file size with the limit set on the server and issue some warnings or positive messages. If the file size of the chosen file is within the server's limit we reactivate our "submit" button of the upload form. (Note that we use jQuery's "on('event')"-functionality to to this.) Otherwise we keep it inactive - until a more suitable file is chosen by the user.

Thus, by very simple means we prevent any unreasonable upload process already before it can be started by the user.

It remains to show an excerpt of the simple PHP target file:

<?php

// start session and output buffer
session_start();
ob_start(); 

$file_size_limit = ini_get("upload_max_filesize");
$ajax_response = array();
$ajax_response['allowed_size'] = $file_size_limit; 

$ajax_response['sys_msg'] .= ob_get_contents();
ob_end_clean();

$response = json_encode($ajax_response);
echo $response;
exit; 

?>

Nothing special to discuss here.

Can we change the limiting parameters during PHP program execution?

No, we can not. But as a developer you may be able to define directory specific settings both for "post_max_size" and "upload_max_filesize" on the server by uploading ".htaccess"-files or ".user.ini"-files to program directories - if this is allowed by the administrator.

The web page php.net/manual/en/ini.core.php shows a column "Changeable" for all important parameters and the respective allowed change mechanisms.
See also:
http://php.net/manual/en/ini.list.php

Different methods of how to change PHP parameters as a user are described here:

 
However, if you are not a developer but a server admin, preventing users from changing PHP ini-paramters may even be more important for you:

 
Enough for today. In the next article of this series

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – VII

we shall have a look at possible problems resulting from timeout limits set for our Ajax transactions.

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – V

In the previous articles of this series about an Ajax controlled file upload with PHP progress tracking

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – IV
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – III
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II
CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I

we have studied elements of an Ajax controlled setup to trigger the upload itself and additional independent jobs for progress tracking.

On the Javascript side we first became acquainted with Control objects [CtrlOs] to fully control HTML elements and the Ajax communication of different forms and objects with the PHP server. We then learned that our main PHP job [handle_uploaded_init_files.php5] triggered from the file upload form is of no use for progress tracking.

In the last article we therefore discussed some Javascript methods to start a sequence of independent Ajax controlled polling jobs. Such jobs can retrieve information about the progress of the file upload from the server periodically and in the background. We learned in addition that we may need some adaption of our polling time interval to the measured data transfer rate. We expect to get the rate information from the server together with some standard progress information as e.g. the sum of bytes uploaded at the moment of polling.

The progress information for the file transfer is continuously updated in the $_SESSION array on the LAMP server according to settings in the php.ini configuration file. In this article we now discuss the rather simple PHP job for fetching the information and sending it back to the client (browser).

The PHP polling job to read progress information and send it to the client

The periodically called PHP job, which we named "check_progress.php5", basically contains four steps:

  • Access the session and start output buffering,
  • instantiate an object which controls information gathering,
  • fetch information from the $_SESSION array,
  • encode and send the JSON object to the Ajax client

PHP program "check_progress.php5"

// start session and output buffer
session_start();
ob_start(); 

// Instantiate progress checker object 
$PC = new ProgressChecker();  

// retrieve and send information 
$PC->check_progr(); 
$PC->encodeAndSendJson(); 
exit; 

// ------------------------------
class ProgressChecker  
{
	//Some variables 
	...
	function __construct() {
		....
	}
	function check_progr() {
		....
	}
	function encodeAndSendJson() {
		....
	}
}

Why did we need the output buffering ob_start()? By using the buffer we control unwanted output produced e.g. by warnings of the PHP interpreter! We want to guarantee that our Ajax client really receives a JSON object (which it expects!) and nothing else. See the third article of this series for more remarks on this point.

Remark, 05.07.2015: We have corrected a mistake in the originally published code. "ob_end_clean()" was removed from the main program code. Note that "ob_end_clean()" must be performed just before the Ajax response is sent. This happens in the method encodeAndSendJson(); see below.

Although not required we put all functionality into a class. This will give us more flexibility in case we need to extend the functionality of the object for progress tracking later on.

Some variables of the class "ProgressChecker"

In our class we first set some initial values of elements of an array "$ajax_response[]". This array will later on be encoded as the JSON response object of our Ajax transaction.

	// The array to build the JSON object from 
	var $ajax_response = array();
	// Some useful variables 
	var $percentage = -1;
	// Error handling 
	var $err = 0;
	// variables to compose the key of progress information in $_SESSION
	var $sess_key_progress 	= '';
	var $key_POST = "progress_key";
	
	function __construct() {
		
		$this->ajax_response['sess_key_progress'] = '';
		
		$this->ajax_response['msg'] = '';
		$this->ajax_response['err'] = 0;
		$this->ajax_response['err_msg'] = '';
		$this->ajax_response['sys_msg'] = '';
		
		$this->ajax_response['finalized'] = 0;
		$this->ajax_response['time'] = '';
		$this->ajax_response['secs'] = 0.0;
		$this->ajax_response['diff_secs'] = 0.0;
		$this->ajax_response['end_time'] = '';
		$this->ajax_response['end_secs'] = 0.0;
	
		$this->ajax_response['bytes'] = -1;
		$this->ajax_response['diff_bytes'] = -1;
		$this->ajax_response['total'] = -1;
		$this->ajax_response['percentage'] = -1;
		$this->ajax_response['rate'] = -1.0;
	
		$this->ajax_response['fs_limit'] = -1;
		$this->ajax_response['max_inp_time'] = -1;
	}

 
We organize the information that may be required by the client: the composed key to access progress information in $_SESSION, some time information and real progress information.

"bytes" represents the number of Bytes received so far on the server, "total" is the total number of bytes of the transferred file (i.e. its size), "percentage" is calculated from "bytes" and "total". In addition we deliver a measured data transfer "rate" in [Bytes/msec]. The reader may compare this with the result handling on the Javascript client side described in the last article of this series.

Last but not least we initialize some variables which shall deliver useful information about parameters set in the "php.ini" file on the server.

You see that we intentionally set some variables to negative values. Such an initialization can help to analyze potential errors on the server and especially after some response has been returned on the client.

Reading progress information from the $_SESSION array

The core of our class is the method check_progress():

function check_progr() {
	
	// Compose the key for the information in the $_SESSION array  
	if (!isset( $_POST[$this->key_POST] ) ) {
		$this->err++;
		$this->ajax_response['err'] = 1;
		$this->ajax_response['err_msg'] .= '<br>The required key was not delivered in $_POST';
	}
		
	if ( $this->err == 0 ) {
		$this->sess_key_progress .= ini_get("session.upload_progress.prefix"). $_POST[$this->key_POST];
		$this->ajax_response['sess_key_progress'] = $this->sess_key_progress;
	}
		
	// Check for progr. information 
		// We have set session.upload_progress.cleanup = Off
		// So we do have some information here
			
	if (!isset($_SESSION[$this->sess_key_progress]) || empty($_SESSION[$this->sess_key_progress])) {
		$this->err++;
		$this->ajax_response['err'] = 2;
		$this->ajax_response['err_msg'] .= '<br>Error: The progress information could not be read from $_SESSION';
		return; 
	}
			
	// Save some information at the first poll for later rate calculations 	
	if (!isset($_SESSION['first_secs'])) {
		$_SESSION['first_secs'] = microtime(true);
	}
	if (!isset($_SESSION['first_bytes'])) {
		$_SESSION['first_bytes'] = $_SESSION[$this->sess_key_progress]["bytes_processed"];
	}
		
	// Get some php.ini parameters
	$file_size_limit = ini_get("upload_max_filesize");
	$this->ajax_response['fs_limit'] = $file_size_limit; 
	$this->ajax_response['max_inp_time'] = ini_get("max_input_time"); 

	// investigate whether file size is too big and return error 
	$file_size = $total / 1024 / 1024; 
	if ( $file_size > $file_size_limit ) {
		$this->err++; 
		$this->ajax_response['err'] = 3;
		$this->ajax_response['err_msg'] .= "<br>Error: The file has a size of " . number_format($file_size, 2, '.', '') . " MByte and is bigger than the allowed limit on the server (" . $file_size_limit . " MByte)"; 
		return; 	
	}
	
	// Retrieve progress information 
			
	// Bytes
	$current = $_SESSION[$this->sess_key_progress]["bytes_processed"];
	$total 	 = $_SESSION[$this->sess_key_progress]["content_length"];
	
	$this->ajax_response['percentage'] = ($current < $total) ? ($current / $total * 100) : 100;
	$this->ajax_response['bytes'] = $current;
	$this->ajax_response['total'] = $total;
		
	$this->ajax_response['time'] = date('%d.%m.%Y-%H:%M:%S');
	$this->ajax_response['secs'] = microtime(true);
	$this->ajax_response['first_bytes'] = $_SESSION['first_bytes'];
	$this->ajax_response['first_secs'] = $_SESSION['first_secs'];
	$this->ajax_response['diff_secs'] = $this->ajax_response['secs']  - $_SESSION['first_secs'];
	$this->ajax_response['diff_bytes'] = $this->ajax_response['bytes'] - $_SESSION['first_bytes'];
	
	// Rate in bytes / msec
	if ($this->ajax_response['diff_secs'] > 1.0 ) {
		$this->ajax_response['rate'] = floor($this->ajax_response['diff_bytes'] / ( $this->ajax_response['diff_secs'] * 1000.0));
	}
		
	// upload finalized ? if yes => cleanup of $_SESSION 
	$done_upl 	= $_SESSION[$this->sess_key_progress]["done"];
	$done_file 	= $_SESSION[$this->sess_key_progress]["files"][0]["done"];
		
	if ($done_upl && $done_file) {
		$this->ajax_response['end_time'] = $this->ajax_response['time'];
		$this->ajax_response['end_secs'] = $this->ajax_response['secs'];
	
		// cleanup of progress variables
		// the real session will be closed by the upload job
		unset($_SESSION[$this->sess_key_progress]);
		unset($_SESSION['first_secs']);
		unset($_SESSION['first_bytes']);
	}
		
}

 
As we already know, we first have to compose the key to access progress information in the $_SESSION array. We must use

  • both information provided by the client to uniquely identify the file transfer process to which our polling job refers
  • and some specific information of the "php.ini" file.

See the third article of this series for more information about this point. Remember that we explicitly took care about the fact that key relevant information reaches the server first - i.e. before the file data.

In the block of program statements we obviously test the existence of progress data - and stop our activities with setting an error flag and an error message in our response array. Such an error situation must be recognized and handled on the client side; see the last article of this series.

Then we add some initial time information and byte information to new elements of the $_SESSION array. Obviously, this happens only at the first polling job. This stored information will later on be used for rate estimations.

Important note:

Our simplified approach to rate estimation via additional $_SESSION variables will only work if you trigger exactly one upload process at a time. If you triggered several upload jobs in parallel (i.e. before previously started jobs have finalized and within one and the same session) you would need to make all progress information which you add to $_SESSION dependent on the identifier of the transfer process.

We leave this extension to the reader and assume that different upload jobs are started from the client only sequentially.

We now fetch 2 parameter values from the "php.ini" file: for the maximum allowed size of upload files and the maximum allowed input parsing time. The latter value determines how much time can be used on the server to deal with the transferred (POST) data. This is a critical limit about which the client should be informed.

In addition: If we find that the size of the transferred file is too big, we return an error to the client.

Next, we retrieve progress information, calculate the reached percentage of transferred bytes and estimate a transfer rate. To avoid any problems with divisions by zero we deliver a rate value only after at least a second (i.e. the default the update time interval on the server for progress information) has passed. Remember that these rate values are used on the client side to adapt the polling period to reasonable values.

Finally, we check by some criteria whether the transfer job has already finished and all file data are received. If this is the case, we erase progress related information from the $_SESSION array. This is absolutely necessary if you want to start further upload jobs during the same PHP session.

Important note:

We are only able to retrieve the information about the end of the file transfer if the progress information is not automatically erased from the $_SESSION array by the tracking engine of the server itself. Otherwise our last polling job would almost always run into problems. So, for our Ajax controlled polling to work flawlessly, we must set:
session.upload_progress.cleanup = Off
in our "php.ini"-file - and do some cleanup ourselves.

At this point I also want to remind the reader that our main job for dealing with the uploaded file data may already have started before the last polling job is triggered. Due to the length of polling time interval such a situation is very probable. We had to take care of this situation on the Javascript client side to avoid a confusing mixture or overwriting of returning messages from both PHP jobs; see the last article of this series for more remarks on this point.

Encoding the JSON object

The final step is trivial:

function encodeAndSendJson() {
	$this->ajax_response['sys_msg'] .= ob_get_contents(); 
	ob_end_clean();
	$response = json_encode($this->ajax_response);
	echo $response; 
}

Some example output of the Ajax controlled client/server interaction for the file upload

After all this theory let us now have a brief look of output of the Ajax interaction between the client and the server. Initially we give the user a chance to select a file

upl_form_1
 
The PHP job that created the web page which contains our upload form also started the required PHP session. You may ignore the upper parts of the displayed excerpt of our web page. The interesting things happen in the lower form which allows for file selection.

The attached information elements (progress bar, small text areas) below actually belong to a separate and different web page area (DIV container) which contains an invisible form for the control of the progress polling jobs.

Note that in the displayed test case we have chosen a ZIP file container (which actually contains 5 files with 5 million records each). Then we start the Ajax controlled transfer with a click on the button "Start Upload". This job starts the file transfer and the first Ajax transaction. This will trigger the main PHP job "handle_uploaded_init_files.php5" for file processing when the data transfer has finalized.

upl_form_2

In parallel the sequence of polling jobs is automatically started in the background. Each Ajax transaction in the time loop triggers a job "check_progress.php5" on the server, which retrieves progress information from $_SESSION as discussed above. The information of each PHP polling job returned to the client via a JSON object is used to enlarge the progress bar and to display additional information text about the progress.

upl_form_3
upl_form_4

Note the display of the bytes transferred, the time elapsed and the remaining time estimated from the measured rate.

After all file data have been received at the server we display some final information in green on the left side. After that the main PHP job (triggered by our original form submit) takes over and the status bar runs again - this time indicating the processing of the individual files we moved to the server in our Zip file container.

upl_form_6

We turn to the handling of the ZIP container and its contents in one of the next articles of this series.

Adaption of the polling interval

In our previous example the polling period stayed at the lowest boundary of 1200 msec. The file (around 80 MByte) was not big enough for an adjustment at the given rate. However, a different example with a bigger file around 150 MByte shows the adaption of the polling period quite clearly. Excerpts from respective messages in the console.log:

time = 1433754422037, 
polling_period = 1200

Present upload percentage = 3.0057897183539
rate is 1485, poll_soll = 1501,
polling_period = 1440

Present upload percentage = 5.0072492005327
rate is 1341, poll_soll = 1652  
polling_period = 1652

Present upload percentage = 7.0087017625639
rate is 1235, poll_soll = 1785
polling_period = 1652

Present upload percentage = 8.0094300619559
rate is 1209, poll_soll = 1821
polling_period = 1821

Present upload percentage = 11.011615536811
rate is 1250, poll_soll = 1765
polling_period = 1821


Present upload percentage = 12.012343836203
rate is 1219, poll_soll = 1807
polling_period = 1821

Present upload percentage = 13.013072712274
rate is 1193, poll_soll = 1844
polling_period = 1821

Present upload percentage = 14.013802741702
rate is 1172, poll_soll = 1875
polling_period = 1821

Present upload percentage = 16.015263377239
rate is 1231, poll_soll = 1790 
polling_period = 1821

Present upload percentage = 17.01599225331
rate is 1211, poll_soll = 1818
polling_period = 1821

Present upload percentage = 18.016722859418
rate is 1193, poll_soll = 1844
polling_period = 1821

Present upload percentage = 19.017454042205
rate is 1177, poll_soll = 1868
polling_period = 1821

Present upload percentage = 21.018911217668
rate is 1206, poll_soll = 1825
polling_period = 1821

 
The rate changes from 1200 msec to 1821 msec in 4 steps. That is exactly what we wanted to achieve.

However, there is a glitch of one additional percent every 4th step. This can be explained by the fact that our polling interval allows for around 1.2 % effective change but that we pick up only a 1% change due to the servers settings between 2 polling events. As the update period on the server is a bit smaller and thus a bit asynchronous to our polling events a glitch is bound to happen around every 4th event on the server. The reader may sketch the polling events and boundaries of the update period on the server in a drawing to get a clear understanding. We can ignore the glitch for all practical purposes.

Note that for larger files at the same rate the adaption would take more steps - but the behavior would more or less be the same.

A first conclusion

So far, so good. Obviously, it is possible to use the progress tracking of PHP versions ≥ 5.4 in combination with a series of Ajax controlled polling jobs. The Javascript side was a bit tricky - but we got it working - and we now have full control about the upload process and its messages. At least in principle ...

However, there are still some severe issues. Especially, if the upload file is big and the rate is small. Then our mechanism may run into trouble due to parameter settings of the PHP engine. In the next article of this series

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – VI

we will therefore discuss how we can get some control over situations in which violations of server limits become probable.