Welcome, Guest. Please login or register.
Did you miss your activation email? July 20, 2008, 04:03:24 AM
Did you miss your activation email? July 20, 2008, 04:03:24 AM
Developers' Blog
Database Changes in SMF 2.0 - Part 2
Righty,
Thought that with a few minutes time on my hand I’d make a bit of a post about the recent changes we’ve made to 2.0 – specifically to the database engine (99% of the forum population may wish to turn away now to save themselves five minutes of boredom).
Right, I’ll start with the motivation behind this. There are two real motivations, one of them is security and the other improved abstraction. In my mind security has been the real driver for this and as such I’ll explain why. You see, there are two main ways of someone “exploiting” web based software (Or certainly we’ve had no others ever affect SMF), they are:
1) SQL Injection attacks – where through some means someone is able to run arbitrary SQL commands on your box, revealing passwords, destroying data etc.
2) XSS (Cross Site Scripting) – where through injection of javascript (Or otherwise) a user may be able to steal your session and “take over” your account.
SMF obviously has always protected against both of these and continues to do so. Whilst both can have major implications in general SQL attacks are the ones that, in my mind, are the cause for most concern. So, how did SMF used to protect against these, well – in two ways:
1) Strings are “escaped” (the means by which variables are made safe) before they are inserted into a database. As the normal source for an injection is malicious data coming into the script from a URL/Form etc SMF used to take the approach of “escape everything” whereby anything coming into the script was made safe – and we’d only make it “dangerous” if we needed to do some manipulation of it – and would make it safe again afterwards.
2) The database query function did some additional checks to ensure if someone *did* manage to exploit SMF that they were limited into the extent of the damage they could cause. This is what I’d call the “second line” of defence – or damage limitation.
When developing 2.0 we’d started to become concerned about our approach to the first of these protection mechanisms. SMF 2.0 has generally become cleverer, more complex. We do more things with incoming variables before we put them in the database and in general that meant making them dangerous more and more often. Every time you make them “dangerous” there is a risk that you forget to make them safe again. We decided that this approach was no longer appropriate as it was beginning to increase as opposed to decrease the security risks in 2.0. So – like all good developers – we changed it.
Compuart, the clever chap that he is, came up with a new scheme for doing this security. The problem with our old security model was all the security was on the “gates”. Once something was “in” SMF it was assumed to be secure – we wanted to change this. Now, instead of the security being on the gates it stands directly in front of the database itself. You can no longer talk to the database without doing through Bertie the Bouncer (Or whatever he is called). To get something into (Or for that matter out of) the database you need to tell it exactly what it is you’re doing – and he’ll make it secure for you.
So, now if I want to set my name to ‘Grudge – King of the Newts’ when I send it to the database I also tell it ‘It’s a string’. If it’s not a string security won’t let it if. Also, security – knowing how dangerous strings are – will clean it up before it gives it to the database. The point is that we can do whatever we want to variables within SMF – I can feel with all kinds of dangerous things – but when it comes to feeding it into the database it takes care of security there and then. Now, I’ll move on from the silly examples and go into a it of practical stuff – those who vomit at the site of a it of PHP best look away now.
So, here’s the old way to update my name: (If you’re confused by the $smfFunc stuff see one of my previous blogs on databases)
Now, things would look quite different:
So, let’s look at things bit by bit:
1) __FILE__ and __LINE__ are gone. We know use backtracing functions in the case of an error. It doesn’t work properly on very old versions of PHP so debugging won’t be quite so hot on 4.3.1 but we thought that they are in the absolute majority and the new method makes queries much cleaner.
2) No single quotes (‘) within the query. In fact, if I were to put a single quote in the query – anywhere – you’d get a big error message.
3) Instead the place the string is meant to go has an inject name which references the key of the array passed as the next parameter. When the query is processed every item in the array is reinserted into the query before being passed to the database. Variables are cast and cleaned as required. This means if you inserted an int it will be cast as an int. If you insert a string it will be escaped and surrounded by quotes.
Another examples:
As for the first example, the string $new_location is tested to be a string, escaped, surrounded with quotes and fills the gap of {string:new_location}. Meanwhile the other type array_int looks at the array element of my_friends, checks it’s an array, check’s all it’s contents are ints, then explodes into a list of numbers into the query.
Of course it’s much easier to see this by actually viewing the code in real examples but I’m hoping this will help. So, how will this change impact people?
Well – for “normal” users – you won’t see a thing. Everything will work as usual – but you should benefit from adding security that will (hopefully) also improve the security of mods as mod authors now can rely on SMF for their database security and not have to think too much about their own protection for SQL issues.
For “mod” users things aren’t too much of a problem. They simply remove (I suspect) all calls to escaping functions like addslashes etc that they currently use. They need to rewrite their queries (In addition all inserts should now use a new function db_insert but I’ve spoken about that before) – but it’s not *that* much work as all mods need to be rewritten for 2.0. Besides which – even though Compuart wrote a great tool for converting the main SMF files I still spent about 6 days over Christmas checking, and in many cases rewriting, every query in SMF so no-one can moan to me about the length of the task as I’ve done it first hand
The main people this will impact is people who integrate SMF deeply within their site. People who use SMF for integrations (For example including SSI.php) and then doing their own queries need to be aware that variables like $_POST and $_GET are no longer escaped by SMF. This means if you previously had this code.
In the past that would have worked. Now that will create a very dangerous security hole. Therefore anyone using SMF for an integration needs to make one of three decisions, either:
1) Convert their queries to use the SMF query function (Highly recommended)
2) Escape $_GET, $_POST type variables individually before they are used in their queries
3) Escape all $_GET, $_POST type variables straight after including SSI.php by doing something like:
I really wouldn’t however recommend doing this. Certainly if you ever use any SMF functions *after* the latter example there is a risk that data will be “double escaped” resulting in lots of slashes appearing in posts, member names etc unintentionally.
There are a few more things about the system but I’ve covered off the bulk of it here. I very much believe that these changes will benefit everyone, particularly users who should always be looking for better and better security. We should be upgrading our site here this weekend so I’ll soon be able to report first hand how easy or otherwise converting an integration is
Grudge
Thought that with a few minutes time on my hand I’d make a bit of a post about the recent changes we’ve made to 2.0 – specifically to the database engine (99% of the forum population may wish to turn away now to save themselves five minutes of boredom).
Right, I’ll start with the motivation behind this. There are two real motivations, one of them is security and the other improved abstraction. In my mind security has been the real driver for this and as such I’ll explain why. You see, there are two main ways of someone “exploiting” web based software (Or certainly we’ve had no others ever affect SMF), they are:
1) SQL Injection attacks – where through some means someone is able to run arbitrary SQL commands on your box, revealing passwords, destroying data etc.
2) XSS (Cross Site Scripting) – where through injection of javascript (Or otherwise) a user may be able to steal your session and “take over” your account.
SMF obviously has always protected against both of these and continues to do so. Whilst both can have major implications in general SQL attacks are the ones that, in my mind, are the cause for most concern. So, how did SMF used to protect against these, well – in two ways:
1) Strings are “escaped” (the means by which variables are made safe) before they are inserted into a database. As the normal source for an injection is malicious data coming into the script from a URL/Form etc SMF used to take the approach of “escape everything” whereby anything coming into the script was made safe – and we’d only make it “dangerous” if we needed to do some manipulation of it – and would make it safe again afterwards.
2) The database query function did some additional checks to ensure if someone *did* manage to exploit SMF that they were limited into the extent of the damage they could cause. This is what I’d call the “second line” of defence – or damage limitation.
When developing 2.0 we’d started to become concerned about our approach to the first of these protection mechanisms. SMF 2.0 has generally become cleverer, more complex. We do more things with incoming variables before we put them in the database and in general that meant making them dangerous more and more often. Every time you make them “dangerous” there is a risk that you forget to make them safe again. We decided that this approach was no longer appropriate as it was beginning to increase as opposed to decrease the security risks in 2.0. So – like all good developers – we changed it.
Compuart, the clever chap that he is, came up with a new scheme for doing this security. The problem with our old security model was all the security was on the “gates”. Once something was “in” SMF it was assumed to be secure – we wanted to change this. Now, instead of the security being on the gates it stands directly in front of the database itself. You can no longer talk to the database without doing through Bertie the Bouncer (Or whatever he is called). To get something into (Or for that matter out of) the database you need to tell it exactly what it is you’re doing – and he’ll make it secure for you.
So, now if I want to set my name to ‘Grudge – King of the Newts’ when I send it to the database I also tell it ‘It’s a string’. If it’s not a string security won’t let it if. Also, security – knowing how dangerous strings are – will clean it up before it gives it to the database. The point is that we can do whatever we want to variables within SMF – I can feel with all kinds of dangerous things – but when it comes to feeding it into the database it takes care of security there and then. Now, I’ll move on from the silly examples and go into a it of practical stuff – those who vomit at the site of a it of PHP best look away now.
So, here’s the old way to update my name: (If you’re confused by the $smfFunc stuff see one of my previous blogs on databases)
Code: [Select]
$smfFunc[‘db_query’](‘’, “
UPDATE {$db_prefix}members
SET member_name = ‘grudge’”, __FILE__, __LINE__);
Now, things would look quite different:
Code: [Select]
$smfFunc[‘db_query’](‘’, ‘
UPDATE {db_prefix}members
SET member_name = {string:name}’,
array(
‘name’ => ‘grudge’,
)
);
So, let’s look at things bit by bit:
1) __FILE__ and __LINE__ are gone. We know use backtracing functions in the case of an error. It doesn’t work properly on very old versions of PHP so debugging won’t be quite so hot on 4.3.1 but we thought that they are in the absolute majority and the new method makes queries much cleaner.
2) No single quotes (‘) within the query. In fact, if I were to put a single quote in the query – anywhere – you’d get a big error message.
3) Instead the place the string is meant to go has an inject name which references the key of the array passed as the next parameter. When the query is processed every item in the array is reinserted into the query before being passed to the database. Variables are cast and cleaned as required. This means if you inserted an int it will be cast as an int. If you insert a string it will be escaped and surrounded by quotes.
Another examples:
Code: [Select]
$new_location = “England’s Home Land”;
$smfFunc[‘db_query’](‘’, ‘
UPDATE {db_prefix}members
SET location = {string:new_location}
WHERE id_member IN ({array_int:my_friends})’,
array(
‘new_location’ => $new_location,
‘my_friends’ => array(3,6,7,3),
)
);
As for the first example, the string $new_location is tested to be a string, escaped, surrounded with quotes and fills the gap of {string:new_location}. Meanwhile the other type array_int looks at the array element of my_friends, checks it’s an array, check’s all it’s contents are ints, then explodes into a list of numbers into the query.
Of course it’s much easier to see this by actually viewing the code in real examples but I’m hoping this will help. So, how will this change impact people?
Well – for “normal” users – you won’t see a thing. Everything will work as usual – but you should benefit from adding security that will (hopefully) also improve the security of mods as mod authors now can rely on SMF for their database security and not have to think too much about their own protection for SQL issues.
For “mod” users things aren’t too much of a problem. They simply remove (I suspect) all calls to escaping functions like addslashes etc that they currently use. They need to rewrite their queries (In addition all inserts should now use a new function db_insert but I’ve spoken about that before) – but it’s not *that* much work as all mods need to be rewritten for 2.0. Besides which – even though Compuart wrote a great tool for converting the main SMF files I still spent about 6 days over Christmas checking, and in many cases rewriting, every query in SMF so no-one can moan to me about the length of the task as I’ve done it first hand
The main people this will impact is people who integrate SMF deeply within their site. People who use SMF for integrations (For example including SSI.php) and then doing their own queries need to be aware that variables like $_POST and $_GET are no longer escaped by SMF. This means if you previously had this code.
Code: [Select]
$_POST[‘test’] = ‘test\’s’;
require_once(‘SSI.php’);
mysql_query(“update smf_members SET member_name = $_POST[test]”);
In the past that would have worked. Now that will create a very dangerous security hole. Therefore anyone using SMF for an integration needs to make one of three decisions, either:
1) Convert their queries to use the SMF query function (Highly recommended)
2) Escape $_GET, $_POST type variables individually before they are used in their queries
3) Escape all $_GET, $_POST type variables straight after including SSI.php by doing something like:
Code: [Select]
require_once(‘SSI.php’);
$_GET = escape__recursive($_GET);
… etc for $_POST
I really wouldn’t however recommend doing this. Certainly if you ever use any SMF functions *after* the latter example there is a risk that data will be “double escaped” resulting in lots of slashes appearing in posts, member names etc unintentionally.
There are a few more things about the system but I’ve covered off the bulk of it here. I very much believe that these changes will benefit everyone, particularly users who should always be looking for better and better security. We should be upgrading our site here this weekend so I’ll soon be able to report first hand how easy or otherwise converting an integration is
Grudge
Loading...
I see can it taking a bit of time to get used to, but overall it seems very positive.
LMAO, Bertie the Bouncer.
int
string
text
array_int
array_string
date
float
identifier
raw
In addition {prefix}, {query_see_board}, {query_wanna_see_board} are all magically replaced with $db_prefix, $user_info['query_see_board'], $user_info['query_wanna_see_board'] respectfully.
Is there a way to find out whether a certain mysql command would be supported.
Obviously I want to avoid functions that are mysql only in my current mods, to avoid making things harder when it comes to update for 2.0.
Specifically the command I'm referring to is INSERT SELECT.
http://dev.mysql.com/doc/refman/5.0/en/insert-select.html
It is supported by PostGreSql.
Pretty much the same as php resource types in terms of type casting. (well, it is the same... duhh *slaps self*)
INSERT SELECT has saved me 1000's of queries in a mod i'm working on.
I forgot about this while making my mods work with the latest SMF beta on my site tonight. I did a work around which I would not suggest users do, but it was a dirty fix to get the mod to install.
I plan on making my mods use the new method that the developers are working on. I like this new method. It is very clean, and infact.. {int:user_id} is much more clear than ' . $user_info['id'] . ' and infact it is nicer.
$smfFunc['db_query'](',',Seems pretty nice and cool."SELECT ID_MEMBER, realName FROM {$db_prefix}members WHERE ID_MEMBER = {int:user_id}",
array(
'user_id' => $user_info['id']
)
);
As for MySQL commands. SMF 2.0 does not actually *police* commands - so you can use MySQL specific ones in queries - they will just obviously fail in PostgreSQL which would be a shame. I don't however think we will enforce mods being generic
Selects are generally easier as they have less variables to insert, below is the query for loading a meber:
$request = $smfFunc['db_query']('', '
SELECT mem.*, IFNULL(a.id_attach, 0) AS id_attach, a.filename, a.attachment_type
FROM {db_prefix}members AS mem
LEFT JOIN {db_prefix}attachments AS a ON (a.id_member = {int:id_member})
WHERE mem.id_member = {int:id_member}
LIMIT 1',
array(
'id_member' => $id_member,
)
);
And for interest, here's an example of inserting data
$result = $smfFunc['db_insert']('ignore',
'{db_prefix}sessions',
array('session_id' => 'string', 'data' => 'string', 'last_update' => 'int'),
array($session_id, $data, time()),
array('session_id')
);
Oooohh, $smfFunc['db_insert'], that's new!
$topicID = $smfFunc['db_insert_id']('{db_prefix}topics', 'id_topic');
Can I ask what we should use instead of mysql_num_rows()?
For the most part you can take any mysql_* or db_* function and it'll follow the $smfFunc['db_*'] format. I can't think of any exceptions off the top of my head.
Mod authors and people who plan to integrate SMF 2.0 via SSI etc. should definitely check this out when SMF 2.0 comes out.
I've converted some stuff to the latest version and it uses a lot of queries with strings, so I just left them as they were, just replacing single quotes with double quotes, and double quotes with single quotes.
Something like,
$smfFunc['db_query']('', 'SELECT something FROM {db_prefix}hello WHERE stuff="' . $yo . '"');I think it does work, though...
$smfFunc['db_query']('', 'SELECT something FROM {db_prefix}hello WHERE stuff={string:yo}', array('yo' => $yo));and then not have to worry about it.As for $smfFunc['db_insert'], how long has that been there for? I never noticed it
Since SMF 2.0 Alpha, if I recall correctly. It's been there for quite a while now, in any case.
I've been using SMF 2.0 since the Alpha, but never went through the abstraction code in great detail...
And I do check most of my pages regularly.
I'll try to change these to the new format anyway, progressively at least.
I will also have to make sure my strings are not escaped before they're passed to db_query... Or maybe I should just use mysql_query for these, I don't know.
And when you need speed, you can use the "security_disable" flag, or just rely on mysql_query (if you're using MySQL
Apart from that $smfFunc['db_query']('you_bet') and then $smcFunc['db_query']('my_ass'), of course..........
Now it's just a shame I can't do the actual upgrade for a little while due to the amount of traffic I'm getting at the mo.
I have to say it's going to be a bit scary when you upgrade your site to SMF 2.0. I think it will be the first proper comparison of 2.0 vs 1.1 performance as your site probably runs up to 99% capacity so if SMF 2.0 is only 5% slower than 1.1 it's really going to show.
Will certainly be interesting
What would you do if it turns out there is a (significant) performance regression between 1.1x and 2.0?
On a more serious note, is there anything in particular which makes you (or Grudge, since he's the one who seemed a bit concerned) worry about 2.0's performance?
and ontopic: I spotted that demonoid.com started a forum and use SMF for it. It's a very dynamic one, >180k posts with more 100k users in less then 3 month (started 14 nov). Using 1.1.4 so when the time came for upgrading to 2.0 they also will have an idea about speed and performance progress or regress
regards
Sadly, i don't see that changing anytime soon
Thanks
Birger
$chatrequest = db_query("
SELECT COUNT(*) AS numb
FROM fc_connections
WHERE userid IS NOT NULL", __FILE__, __LINE__);
list ($chatcount) = mysql_fetch_row($chatrequest);
mysql_free_result($chatrequest);
if ( $chatcount == "1" ) {
$singularplural2 = "";
} else {
$singularplural2 = "ä";
}
$chatrequest = $smcFunc['db_query']('', '
SELECT COUNT(*) AS numb
FROM fc_connections
WHERE userid IS NOT NULL');
list ($chatcount) = $smcFunc['db_fetch_row']($chatrequest);
$smcFunc['db_free_result']($chatrequest);
if ( $chatcount == "1" ) {
$singularplural2 = "";
} else {
$singularplural2 = "ä";
}
I have another one
// array for the member ID's
$team = array();
// get all members of group 1 (administrators)
$group = 2;
$res = db_query("SELECT ID_MEMBER
FROM members
WHERE ID_GROUP = $group ", __FILE__, __LINE__);
while ($row = mysql_fetch_assoc($res))
$team[] = $row['ID_MEMBER'];
mysql_free_result($res);
global $smcFunc;
$chatrequest = $smcFunc['db_query']('', '
SELECT COUNT(*) AS numb
FROM fc_connections
WHERE userid IS NOT NULL');
list ($chatcount) = $smcFunc['db_fetch_row']($chatrequest);
$smcFunc['db_free_result']($chatrequest);
if ( $chatcount == "1" ) {
$singularplural2 = "";
} else {
$singularplural2 = "ä";
}
Try that (same code as what thantos posted, but with $smfFunc globaled.
// array for the member ID's
$team = array();
// get all members of group 1 (administrators)
$group = 2;
$res = $smcFunc['db_query']('', 'SELECT id_member
FROM {db_prefix}members
WHERE ID_GROUP = {int:group}',
array(
'group' => $group,
)
);
while ($row = $smcFunc['db_fetch_assoc']($res))
$team[] = $row['id_member'];
$smcFunc['db_free_result']($res);
Edited: Fixed mistake causing it not to work.
Second one:
Fatal error: Function name must be a string in /var/www/fs1/0/public_html/foorumi/Sources/Load.php(1857) : eval()'d code on line 45