62 comments

[ 2.7 ms ] story [ 120 ms ] thread
Any word on when all functions will be UTF-8 safe?
Other than "substring", how often do you really run into this problem? I have never had a problem with PHP and UTF-8.

Oh, good luck trying to match all unicode punctuation with Python/Javascript. With php it is as simple as preg_match('/\pP/u'

I deal with a lot of UTF-8 issues, and while php is super ugly, there are pretty good solutions in cases where it requires ugly hacks in other languages.

strlen and strpos are two fairly commonly used functions that come to mind.
Pretty often. Yes you can use mb_* functions everywhere but not all string functions have an mb_* counterpart, and forget just once and you risk blowing up the entire rest of the request. Not to mention times when you have to use a 3rd party library that doesn't bother with mb_*.

Native UTF is such an important thing... I know it's tough to implement, but come on guys!

I do run into this fairly regularly. The mb_ functions are solid, but I was hitting something the other day where a character, I think it was NBSP, was causing a string to output empty in 5.4 (but was working fine in 5.3).

I think there's something that needs to be fixed at a core level, maybe with PHP 6, that just guts how the language deals with multibyte. Even if it means making it backwards incompatible. I'd probably take the opportunity to drop the mb_ functions, namespace the entire language, and make it multibyte by default. Needs doing eventually!

It's a common misconception. Many people don't understand that the normal string functions are perfectly safe on UTF-8, as long as you don't use hardcoded lengths or offsets. I.e. substr($str, 0, 50) is not safe due to the explicit "50" in there, but substr($str, 0, strpos($str, "foo")) will work correctly on any well-formed UTF-8.

If people have encoding issues in PHP it usually just means that they didn't manage to set up their database properly (you know, finding which one of the 10 encoding options in MySQL is the right one ;)

From my personal experience I've had a lot more issues with encoding in Python than I had in PHP - exactly because PHP ignores encoding and lets me deal with it.

I agree. Python creates a lot of problems that are non-obvious. Like os.walk('.') works just fine, right up to the point where it silently trashes all unicode file names.
PHP should keep everything as-is and simply add new data type for unicode strings. It should be utterly and completely incompatible with any current function that accepts strings:

    $binString = "hello";
    $unicodeString = u"Hello";
    strlen($binString); // 5
    strlen($unicodeString); // Error
And then developers can slowly start making functions and methods more unicode aware as necessary. Or make an entirely new string API. Then just have functions to take binary strings and convert them to unicode strings (providing an encoding) and unicode strings to binary strings (also providing an encoding).

This would be way safer and simpler than PHP6 or Python 3.

As I understand in PHP to work with Unicode you need to use Multibyte String Functions since Unicode encodes into 2+ bytes vs the traditional 1 byte. So doing a "substr" as per your example would not work unless you use "mb_substr".
Sorry, but PHP's string handling is abysmal.

- strlen can't be used for length checks if you care about length in characters, a problem compounded when you use a real database where field lengths are defined in characters instead of bytes

- the only way to iterate char by char instead of byte per byte is to use mb_substr, as there is no trick to make $str[$i] do anything but return bytes.

- the string and array API's have inverse ordering of their parameters, which means that after a decade of writing PHP I still can't remember which is which without looking it up

- Typing mb_ in front of everything is ugly enough, but it also makes autocompletion tricky, especially since the strings don't have methods. (e.g. $str->pos())

- Speaking of which, since the strings don't have methods, they stick out like a sore thumb in OO code. String and array handling code invariably ends up ugly unless you write everything procedural style (and then you have other issues).

- Sort() cannot be made to sort unicode on windows, regardless of which parameters you give it. In fact, the only way to sort unicode on windows is by using the Collator from the intl extension. Part of that is microsoft's fault by not supporting UTF-8 in the windows API's at all, but PHP isn't helping.

- If you don't care about windows, the proper way to sort text is first calling setlocale(LC_COLLATE, "en_US.UTF8") and then passing the SORT_LOCALE_STRING argument as second parameter to every call to sort(). Ugly, ugly, ugly.

- natsort(), aka "natural sort" cannot be used to sort text like a human would expect, in any context. It always produces invalid results, even for ANSI codepages. (e.g. try to sort resume, rope and résumé)

- The use of utf8_decode and utf8_encode is actively harmful in almost all circumstancces. There is never a good reason to use them, since the very rare case where you need them iconv or mb_convert_encoding are better suited. Yet, the PHP documentation doesn't tell you this, causing lots of people to be led astray (as I once was).

- Oh yeah, and there are no less than three API's for unicode string handling, the mb_ functions, the iconv_ functions and the grapheme_ functions. What's the difference? I don't know, and I really can't be bothered to read PHP's source to find out.

- htmlentities() always requires the parameters ENT_QUOTES, "UTF-8" to do its job securely (well, almost, as it doesn't encode forward slash which OWASP recommends). Unless you use a wrapper, your code is yet again uglified.

- The secure way to JSON-encode text is, and I kid you not, json_encode($data, JSON_HEX_TAG|JSON_HEX_APOS|JSON_HEX_QUOT|JSON_HEX_AMP). Try typing that three times in a row, I dare you.

- And finally, mysql is by far the worst database for unicode handling, because it cannot sort unicode text according to the standard, at all, no matter what you do. That's not PHP's fault ofcourse, but since I'm bitching... :)

The json_encode one is particularly annoying as you can't even consider sending JSON to the browser and using it in JS without all the flags on. especially if you're dealing with user data or scraping websites.
Just out of curiosity, could you expand on that a little more? I'm currently in the process of ripping my hair out with json_encode issues.

We send back a json_encoded array containing a property "html" with html markup to be injected back into a contenteditable div. Originally I had a ton of problems if there happened to be the invisible unicode character in there (php's str functions could not find/replace this character no matter what I tried). However, this is the first I've seen those flags the GP mentioned.

If you look up the json_encode function on php.net it lists the flags. As I understand it there's a difference between the encoding allowed in javascript and the encoding allowed in JSON which means certain characters which would be valid in a javascript object don't necessarily work in JSON, so you end up having to escape everything possible. There's one flag (JSON_ESCAPE_UNICODE) which doesn't work on the version of PHP I develop on so I ended up applying code listed in the comments. This also adds some cruft similar to what google/facebook etc use if you want it. I haven't had any issues with it yet but, being what it is, there may be a more elegant way to accomplish it.

   function AsJSON($arr, $cruft=null){
		
	//convmap since 0x80 char codes so it takes all multibyte codes (above ASCII 127). 
	//So such characters are being "hidden" from normal json_encoding
        array_walk_recursive($arr, function (&$item, $key) {  	
        	if (is_string($item)){  
        		$item = mb_encode_numericentity($item, array (0x80, 0xffff, 0, 0xffff), 'UTF-8');
        	}
        });
        
        $JSON = mb_decode_numericentity(
        	json_encode($arr, JSON_HEX_TAG | JSON_HEX_APOS | JSON_HEX_QUOT | JSON_HEX_AMP), 
        	array (0x80, 0xffff, 0, 0xffff), 
        	'UTF-8'
        );

        if(json_last_error() === JSON_ERROR_NONE){ 
	  return $cruft.$JSON;
	 	}
	}
You might also consider looking at the ctype_* functions if you haven't.
Seeing the type of content, the character in question might very well be a BOM.

You can identify them by pack("CCC",0xef,0xbb,0xbf), encoding to ""

define('OH_MY_THOSE_FLAG', FLAG | FLAG | FLAG | FLAG)

But then again, my name gives me special powers.

Nevertheless...
Yeah but as with many complains about PHP, how would you define the default behavior? I'm seeing so many people who expect PHP to always do stuff how they want, because of how easy it is to work with. I never read those complaints about super tricky C cases.

Modifying the content by default would be completely counter-intuitive, specially with a function that is used mostly in API and thus being encoded / decoded in different places and languages.

IMO content filtering is specific to the view where you output it, and should be done there. Any hazardous content should always go through a sanitize function when you echo it in the middle of HTML.

(comment deleted)
You're the reason PHP has a bad reputation. You have no idea what you're doing, and you use your mouth not only to breathe, but to spill hate on it too!

80% of your points can be answered by wrappers/helpers. How often do PHP developers have to walk through a user-generated string of multibyte cacarcters? Ok, didn't think so. And when you do you have all the required tools. 99% of string handling is simple copy, and the byte apporach is great at that.

A few points that stand out, and give an amusing idea of how you implement stuff:

    "Unless you use a wrapper"
    "What's the difference? I don't know, and I really can't be bothered to read"
    "Try typing that three times in a row, I dare you"
    "Oh yeah, and there are no less than three API"
    "a real database where field lengths are defined in characters"
    "makes autocompletion tricky" on native function...
PS:

- natsort() is intended for numbers

- ENT_QUOTES isn't required unless you use single quotes to encapsulate your outputs

- ugly code = using different function names.. ok

- PHP isn't helping Microsoft.. ok

- utf8_decode is a shortcut for iconv() using the 2 most common encodings, just FYI

Edit: formatting

I agree with most of your bashing. PHP is a fucking mess, and non OO strings is a hangover vomit from C.

That being said, when there is a problem, there actually is a solution. PCRE actually works. Javascript, for instance, has no collation support at all as far as I can tell.

Personally, most of the problems with UTF-8 are mixed content issues.

The best fix is not the Python route, but rather just deprecating a bunch of stuff such as utf8_encode/decode. Throw a warning when any database connection is not utf8. Throw a warning when the OS is not setup to return UTF-8. It is more important that people run php in a end-to-end utf-8 environment, than changing the internals. Once people have a good environment, they will stop talking about strlen/strpos which are really not much of a problem. Maybe they should be renamed bytelen/bytepos, but php has too many of that type of problem to count.

99% of the UTF-8 problems don't exist if everything is UTF-8. Counting unicode code points vs bytes is not the real problem. The real problem is bullshit like 'SET NAMES utf8' / setlocale('LC_ALL','en_US.utf-8')

BTW, what language do you think gets this stuff right? Go looks promising, but it is brand new. I have problems with pretty much every language I know well: javascript/python/Objective-C/PHP

i would love to see them make an explicit BC break and fix everything listed at http://phpsadness.com/
If you break BC you might as well just another language, because that's what it will be.

The correct approach is slow and steady improvements. I suspect standard library improvements will come not by renaming existing functions but by adding new API.

The Perl way is to declare minimum language version to execute a code file. E.g. use v5.12;

Then you can do extensions [even changing central functionality] in a sane way, with backward compatibility. (A language implementation need to implement deprecated features for older versions, of course.)

I agree with your point re APIs.

I've had this idea of minimum language version as well. In PHP it should probably done at the namespace level. It doesn't even need new language syntax, just a function to declare the minimum PHP version for a namespace and child namespaces:

    set_minimum_version('My/New/Component', '5.7.8');
This could be called in the component itself or maybe by a composer autoloader. Then PHP components that expect different version could live together.

The big problem, as you mentioned, is that the interpreter would have to maintain all the deprecated features and behavior for older versions and that would make development and maintenance of PHP much more complicated.

That's what we do with Composer, we set the minimum PHP version.
I've done and seen it done in wordpress plugins, sometimes.
And thousands of developers would hate it. Sure, some things are ugly but the usage is too widespread in order to get rid of them. Look at the register_globals disaster and how long it took to throw that out. And it was only one little thing. Imagine tens or hundreds of them.
Yeah, because that worked out SO well with Python… ;P
It amuses me people want PHP to follow the path of Python or Perl. On the latest version of Ubuntu: PHP 5.5, Python 2.x Perl 5.x. PHP has gradually improving, adding functionality, and spreading minimal BC breaks over a long period. If PHP had went the PHP 6 route back round ~08, it'd likely be split into two projects already, splitting up the community with it.
It happened with PHP4 -> PHP5 already. That was a huge split.
Sure PHP had a lot of catch-up to do, but they're really on a fast pace these days.
In ten years it might actually be a first-class language.

It's going to take a ferocious amount of work to shed its reputation as a web-only scripting language.

> shed its reputation as a web-only

What? It is web-only. That's the entire point behind it. It's not a reputation, it's the goal of the project.

php5-cli is pretty robust, to be honest. It is a fairly slick server-side scripting language to integrate processing tasks inside of a normal web environment.
Implemented a websocket server in PHP as a curiosity, which is a php cli daemon independent of apache.

Its absolutely awesome barring having to work with the websocket framing protocol with ordinal casts, etc. PHP has an odd flavor of hex/ binary arithmetic. You pretty much just pull characters out of a string byte by byte and use bit shifting/ ANDing to get a message decoded.

And the best part! No need for anything like APC for the cli; it's useless as the daemon doesn't work like a standard PHP lifecycle in apache.

I did this for a project in 5.3 (client: "PHP EVERYTHING"), it wasn't as difficult as I thought it'd be, but I will admit, there are better solutions.
It's the "better solutions" that make me wonder why people go down this road for any reason other than academic curiosity.

Given an arbitrary back-end written in Python, Ruby, Go, Perl, Java, Scala or PHP I would expect, nine times out of ten, the bad apple to be PHP.

Anyone who knows how to program PHP properly, and by that I mean using all of the modern conventions, would be far more productive in something like Java or Python where they didn't have twenty years of appallingly bad design decisions influenced by its role as a web scripting language weighing down their code.

>Anyone who knows how to program PHP properly, and by that I mean using all of the modern conventions, would be far more productive in something like Java or Python where they didn't have twenty years of appallingly bad design decisions influenced by its role as a web scripting language weighing down their code.

Yes, because Java doesn't have it's own twenty years of bad decisions...

You can build Android apps in Java. It's not pigeonholed.
Also, Symfony 2 makes it almost trivial to build cli applications.
The subtitle for that book could be "Just because you can doesn't mean you should".

Why would anyone write non-web code in PHP? It's completely out of its element.

If your selling point is "not having to learn" you're not being very fair to developers that use PHP. If you can't learn, you've got a problem. New technology comes up all the time, and if you can't adapt, you'll get left behind.

Would you recommend a PHP to JavaScript converter so you can do PHP in the browser and avoid all that painful "learning"?

What about "bash beyond the shell"? It's possible. Would you want to write an application in bash? Please, do not.

Everything that made PHP popular and versatile is now a massive liability. Perl went through the same cycle from saviour to relic and that community has been struggling to try and reinvent itself (Perl 6) in a manner that's in keeping with today's concerns.

The level of change you'd have to introduce in PHP to make it current is so staggering you may as well write a whole new language. Even the relatively modest changes made to modernize Python have caused an enormous rift.

I don't think so. It just piles up all the futures on top of old futures which just makes it more spaghetti-like.

Like their OOP and Namespaces - Sure, cool that PHP has those but compared to other languages like Ruby, C#, Java, etc.. PHP implements these futures horribly.

OOP in PHP is pretty much the same as most other languages. And the implementation of namespaces is literally the only choice when combined with all the other existing PHP features.

If you're looking for mis-features, you need better examples than these.

This is true. However, in my opinion, moving quickly is less important than changing effectively. Effective change can happen quickly or slowly, depending on the change intended and the effort required, but it's a different metric than simply moving in some direction while retaining the previous state.

PHP is painful to work with for a lot of reasons. It's also very easy to work with for a lot of reasons, but in my opinion, a lot of those -- despite being considered by some to be strengths of the language -- are used, and in many cases intended, to be shortcuts to functionality a more robust language would embrace explicitly.

PHP suffers from many, many failures of design. It suffers further, and worse, from an ongoing and intentional failure to address almost all of these failures. It's Frankenstein's language. If it were sentient, it would rebel and run off into some cold and inhospitable place.

A fast pace is velocity. They have that. What they don't have is a good direction, and without a good direction, a long vector is at best a roll of the dice.

Not sure what you are into but PHP is anything but painful to work with. It's the easiest to get started, cheapest to run, and frankly when you actually use it for what it is intended, output web pages, I fail to find better.

At the end of the day, if you keep it simple you can go far without ever needing to do much maintenance and scaling. Most failure to scale I've seen come from bloated code base and code written by people who are not PHP developers.

From a distance PHP is splitting cells. People added tools to its eco system, and practices based on pseudo-Java OOP. See PSR code style that basically forbids PHP to be PHP, no inline formatting, split lib as side-effect free and 'main' .. The people behind this movement will probably have integrated the new language construct very soon. So instead of Frankenstein I see a two-headed organism.
The arguments in the linked email of some Ilia Alshanetsky regarding the shorthand array syntax are incredibly dense...
I'm liking the variadic functions syntax. It's definitely better than requiring all extra parameters to be sent as arrays. Even if that's how they end up in the function.
The current implementation doesn't require extra params as arrays... you can pass as many (or as few) parameters to any PHP function regardless of defined function signature; the declared parameters will be auto-populated. Additionally, you can call func_get_args() inside of any function and get an array with all params, meaning it's a trivial array_slice (and/or list) away from extracting all vargs.

That said, the new ...$params is 100% better.

I really like the ...$param syntax. When we first created NOLOH (http://www.noloh.com) back in 2005 we decided on $dotdotdot as a syntactic salt, but this is obviously much nicer. Also, by ref ... is really really nice. Very few cases where it's necessary, but having that is a pretty big deal.

Since PHP 5.3, PHP has gotten some really nice features, they've definitely been doing a pretty good job over the past few years.

Would REEEEEEALLY like to see readonly properties in classes.

They had a proposal a while ago to add getter and setter function support, which would be cool and would've allowed it (with some function overhead), but is probably a bit OTT for PHP.

readonly properties would allow you to make truly solid classes. You'd be able to lock them down from outside interference.

Methods could trust that property values hadn't been tampered with from outside, while still providing a way for outside code to read those values extremely quickly, without function overhead.

I think it's an awesome idea, especially since the getter/setter proposal failed. It's very PHPy, and it strikes a nice middle ground between fully public properties and private ones.

External code really shouldn't be able to mess with my class's internal properties anyway, it violates separation of concerns.

You can already do this by using a magic setter that blows out an exception whenever its called for whichever variables you denote. A little hacky, but it does indeed prevent setting of certain vars in your classes.
Actually, even easier than SDGT's or krapp's suggestions is to declare your properties private or protected and use a magic getter to expose those properties as public readonly.

It's so easy to do that a readonly keyword seems unnecessary.

I agree with you. This is the closest i've been able to get...

   class whatever{
     
     private $Properties=array();

     function __set($key,$val){
        try{
         if(!isset($this->Properties[$key])){
            $this->Properties[$key]=$val;
         }
        }catch(Exception $e){
        //...
        }
      }

      function __get($key){
       try{
        if(isset($this->Properties[$key])){
          return $this->Properties[$key];
        }
       }catch(Exception $e){
       //...
       }
     }
    }
I just noticed some errors in this but the basic gist works... I shouldn't have added the isset checks, those would prevent it from throwing the exceptions -_-
Your code doesn't work...

The catch will never be called, and you should never use isset() because it will return false if the value exists but is null.

    whatever->param = null;
    whatever->param = 123;
Both lines will execute here.

    function __set($k, $v){
        if (array_key_exists($k, $this->Properties)){
            throw new ReadOnlyException();
        }else{
            $this->Properties[$k] = $v;
        }
    }
Yes, thank you. I noticed the problem after it was too late to edit. Slightly embarrassing.
Does anyone else here find the PHP namespace format horrible and unlike the standard dot notation?
I hated it till I started using it, now I don't mind. Dot was never used in PHP because it's the string concat operator. PHP needs a unique operator for namespaces because the types of symbols (is it a namespace, class, interface, etc) aren't known at compile time (unlike Java, C#, etc).
>Dot was never used in PHP because it's the string concat operator.

So how about them using the -> operator? Most languages that use . for class method access also reuse it for namespacing.