packing and unpacking zero terminated strings in python

July 8th, 2008

so i found a problem with the struct module in python: it cannot pack or unpack zero terminated strings! This is pretty bad if you want to parse or generate data that must conform that zero termination c-string idea. So i wrote some wrapper functions for pack and unpack that add the format ‘z’. z means zero terminated string.

What it does simply is:

  • on packing find and add z zero if necessary. Use the correct string size of the found string argument to pack it correctly
  • on unpacking unpack every format as normal, just manually unpack the ‘z’ character format.

This will enable you to write:

Python [Show Plain Code]:
  1. print unpackExt("z", packExt("z", "test1"))
  2. print unpackExt("z", packExt("z", "test1 test2"))
  3. print unpackExt("z", packExt("z", "test1\0test2"))
  4. print unpackExt("ibzi", packExt("ibzi", 200, 3, "test1\0test2", 34))
  5. print unpackExt("ibzizf", packExt("ibzizf", 200, 3, "test1\0test2", 34, "test string", 0.234))

which will result in:

['test1']
['test1 test2']
['test1']
[200, 3, 'test1', 34]
[200, 3, 'test1', 34, 'test string', 0.23399999737739563]

download the script: struct-zerostrings.py

feel free to use, but please report back if you found problems :)

Posted in python, coding | 1 Comment »

one c macro to rule them all

June 29th, 2008

or how to comment a line using the precompiler: http://www.ddj.com/cpp/184401344

the short version for the impatient:

#ifdef _DEBUG
 #define DEBUGCLASS myClass::getSingleton()
#else
 #define COMMENT SLASH(/)
 #define SLASH(s) /##s
 #define DEBUGCLASS COMMENT
#endif

DEBUGCLASS.dosomething();

Posted in coding | No Comments »

how to fully use maxmind’s geolocation DB with php

April 16th, 2008

or: how to use geolocation usefully. This solution tries to solve i.e. the problem of calculating the distance between two country by simply comparing the continent of each. So you can simply use a fitting server on the user’s same continent or such.

application example: geolocatephp.txt

the included php file (use it as you like) (the data is based on this)
geotoolsphp.txt

Posted in php, coding | 3 Comments »

indentcode.net

March 3rd, 2008

i was really missing the “indent” tool under windows for my c++ source code :-\
and i found no online converter, so i created one in under an hour:


indentcode.net

Posted in coding, website | 1 Comment »

about mysql…

January 28th, 2008

some small notes from myself:
- never use timestamps in queries and use such queries to select them:

SQL [Show Plain Code]:
  1. SELECT * form TABLE WHERE timestamp > $begin AND timestamp < $end;

this will result in an overused CPU ! :-\

- also how to loop over entries in sql via stored procedures in sql:

SQL [Show Plain Code]:
  1. DELIMITER |
  2. DROP PROCEDURE IF EXISTS createstats |
  3. CREATE PROCEDURE createstats()
  4. BEGIN
  5.   DECLARE v_fileid INT;
  6.   declare no_more_rows BOOLEAN DEFAULT FALSE;
  7.   declare cursor1 cursor FOR
  8.   SELECT id FROM files;
  9.   declare continue handler FOR NOT found
  10.   SET no_more_rows = TRUE;
  11.  
  12.   open cursor1;
  13.   LOOP1: loop
  14.     fetch cursor1 INTO v_fileid;
  15.     IF no_more_rows then
  16.       close cursor1;
  17.       leave LOOP1;
  18.     end IF;
  19.  
  20.     INSERT INTO files_tempstats (`fileid`, `dltotal`, `ratecount`, `ratesum`) VALUES (
  21.       v_fileid,
  22.       (SELECT count(`stats`.`value`) FROM `stats` WHERE `stats`.`fileid`=v_fileid),
  23.       (SELECT count(`ratings`.`id`) FROM `ratings` WHERE `ratings`.`fileid`=v_fileid),
  24.       (SELECT sum(`ratings`.`rating`) FROM `ratings` WHERE `ratings`.`fileid`=v_fileid)
  25.     );
  26.   END LOOP LOOP1;
  27. END

Posted in coding, website | No Comments »

great OS-independent Python Editor

January 15th, 2008

Just found the “SPE IDE” = Stani’s Python Editor :)

get it here

Posted in python, coding | No Comments »

YABT (yet another new blog title)

January 8th, 2008

Isaac was so kind to tell me that the meaning of the title wont fit with the amount of number (meaning here).

so i updated it to a more correct form ;)

Posted in website | 1 Comment »

kcachegrind

January 8th, 2008

very useful to see the call Graph and performance of your program:
screenshots: http://kcachegrind.sourceforge.net/cgi-bin/show.cgi/KcacheGrindShot
how to get it (under gentoo):

emerge -av kcachegrind valgrind

How to use:
Collect data program during program execution

valgrind –tool=callgrind –trace-children=yes –num-callers=8 –collect-jumps=yes –instr-atstart=no ./yourprogram

Then open the output file that is created using kcachegrind!

Posted in gentoo, coding | No Comments »

write your own wireshark protocol dissector

January 4th, 2008

Dissectors in Wireshark will convert the pure data of a packet into a custom protocol with readable data, so you can see values of your protocol instead of just binary data.

How is it done?

First open the file “init.lua” in your wireshark directory and comment out this line:

Lua [Show Plain Code]:
  1. disable_lua = false; do return end;

to

Lua [Show Plain Code]:
  1. –disable_lua = false; do return end;

then add to the bottom of the file this line:

Lua [Show Plain Code]:
  1. dofile("ror.lua")

(ror is the protocol in my case)

After that you should create a file named “ror.lua” and copy+paste the following content:

Lua [Show Plain Code]:
  1. – dissects the RoR Protocol
  2. packetnum = -1
  3. ror_proto = Proto("ror","ROR","RoR Protocol")
  4. – create a function to dissect it
  5. function ror_proto.dissector(buffer,pinfo,tree)
  6.     pinfo.cols.protocol = "ROR"
  7.         cmd = buffer(0,4):le_uint()
  8.        
  9.         packetnum = pinfo.number
  10.  
  11.         local subtree_packet = tree:add(ror_proto,buffer(),"Rigs of Rods ("..buffer:len()..")")
  12.        
  13.         local subtree_header = subtree_packet:add_le(buffer(0,12), "RoR Header (12)")
  14.        
  15.         cmd_str = "unkown"
  16.         if cmd == 1000 then
  17.                 cmd_str = "MSG2_HELLO"
  18.         elseif cmd == 1001 then
  19.                 cmd_str = "MSG2_VERSION"
  20.         elseif cmd == 1002 then
  21.                 cmd_str = "MSG2_FULL"
  22.         elseif cmd == 1003 then
  23.                 cmd_str = "MSG2_BANNED"
  24.         elseif cmd == 1004 then
  25.                 cmd_str = "MSG2_WELCOME"
  26.         elseif cmd == 1005 then
  27.                 cmd_str = "MSG2_USE_VEHICLE"
  28.         elseif cmd == 1006 then
  29.                 cmd_str = "MSG2_SPAWN"
  30.         elseif cmd == 1007 then
  31.                 cmd_str = "MSG2_BUFFER_SIZE"
  32.         elseif cmd == 1008 then
  33.                 cmd_str = "MSG2_VEHICLE_DATA"
  34.         elseif cmd == 1009 then
  35.                 cmd_str = "MSG2_USER"
  36.         elseif cmd == 1010 then
  37.                 cmd_str = "MSG2_DELETE"
  38.         elseif cmd == 1011 then
  39.                 cmd_str = "MSG2_CHAT"
  40.         elseif cmd == 1012 then
  41.                 cmd_str = "MSG2_FORCE"
  42.         elseif cmd == 1017 then
  43.                 cmd_str = "MSG2_USER_CREDENTIALS"
  44.         elseif cmd == 1019 then
  45.                 cmd_str = "MSG2_TERRAIN_RESP"
  46.         elseif cmd == 1020 then
  47.                 cmd_str = "MSG2_WRONG_PW"
  48.         elseif cmd == 1021 then
  49.                 cmd_str = "MSG2_RCON_LOGIN"
  50.         elseif cmd == 1022 then
  51.                 cmd_str = "MSG2_RCON_LOGIN_FAILED"
  52.         elseif cmd == 1023 then
  53.                 cmd_str = "MSG2_RCON_LOGIN_SUCCESS"
  54.         elseif cmd == 1024 then
  55.                 cmd_str = "MSG2_RCON_LOGIN_NOTAV"
  56.         elseif cmd == 1025 then
  57.                 cmd_str = "MSG2_RCON_COMMAND"
  58.         elseif cmd == 1026 then
  59.                 cmd_str = "MSG2_RCON_COMMAND_FAILED"
  60.         elseif cmd == 1027 then
  61.                 cmd_str = "MSG2_RCON_COMMAND_SUCCESS"
  62.         end
  63.        
  64.         datasize = buffer(8,4):le_uint()
  65.        
  66.         pinfo.cols.info:set("Command: "..cmd_str)
  67.         subtree_header:add_le(buffer(0,4),"command: ".. cmd_str .." (" .. cmd..")")
  68.         subtree_header:add_le(buffer(4,4),"source: " .. buffer(4,4):le_uint())
  69.         subtree_header:add_le(buffer(8,4),"size: " .. buffer(8,4):le_uint())
  70.  
  71.         databegin = 12
  72.         dataend = databegin + datasize
  73.         local subtree_data = subtree_packet:add_le(buffer(databegin,datasize), "RoR Data ("..datasize..")")
  74.        
  75.         if cmd == 1000 then
  76.                 local info = "RoR Version: "..buffer(databegin, datasize):string()
  77.                 subtree_data:add_le(buffer(databegin, datasize), info)
  78.  
  79.         elseif cmd == 1001 then
  80.                 local info = "RoR Version: "..buffer(databegin, datasize):string()
  81.                 subtree_data:add_le(buffer(databegin, datasize), info)
  82.         elseif cmd == 1002 then
  83.                 local info = "Server full! "..buffer(databegin, datasize):string()
  84.                 subtree_data:add_le(buffer(databegin, datasize), info)
  85.         elseif cmd == 1003 then
  86.                 local info = "user banned! "..buffer(databegin, datasize):string()
  87.                 subtree_data:add_le(buffer(databegin, datasize), info)
  88.         elseif cmd == 1004 then
  89.                 local info = "you are welcome. "..buffer(databegin, datasize):string()
  90.                 subtree_data:add_le(buffer(databegin, datasize), info)
  91.         elseif cmd == 1007 then
  92.                 local info = "buffer size for vehicle data: "..buffer(databegin, datasize):le_uint()
  93.                 subtree_data:add_le(buffer(databegin, datasize), info)
  94.         elseif cmd == 1009 then
  95.                 local info = "deprecated rornet_2.0 packet"
  96.                 subtree_data:add_le(buffer(databegin, datasize), info)
  97.         elseif cmd == 1010 then
  98.                 local info = "deleting user! "..buffer(databegin, datasize):string()
  99.                 subtree_data:add_le(buffer(databegin, datasize), info)
  100.         elseif cmd == 1011 then
  101.                 local info = "chat: "..buffer(databegin, datasize):string()
  102.                 subtree_data:add_le(buffer(databegin, datasize), info)
  103.         elseif cmd == 1012 then
  104.                 local info = "force command (yet not used): "..buffer(databegin, datasize):string()
  105.                 subtree_data:add_le(buffer(databegin, datasize), info)
  106.         elseif cmd == 1017 then
  107.                 subtree_data:add_le(buffer(databegin,20),"nickname: ".. buffer(databegin, 20):string())
  108.                 subtree_data:add_le(buffer(databegin+20,40),"SHA1 password: " .. buffer(databegin+20,40):string())
  109.                 subtree_data:add_le(buffer(databegin+60,40),"SHA1 uniqueID: " .. buffer(databegin+60,40):string())
  110.         elseif cmd == 1019 then
  111.                 local info = "using terrain "..buffer(databegin, datasize):string()
  112.                 subtree_data:add_le(buffer(databegin, datasize), info)
  113.         elseif cmd == 1020 then
  114.                 local info = "wrong password!"..buffer(databegin, datasize):string()
  115.                 subtree_data:add_le(buffer(databegin, datasize), info)
  116.         elseif cmd == 1021 then
  117.                 local info = "rcon login, with SHA1 password:"..buffer(databegin, datasize):string()
  118.                 subtree_data:add_le(buffer(databegin, datasize), info)
  119.         elseif cmd == 1022 then
  120.                 local info = "rcon login failed: "..buffer(databegin, datasize):string()
  121.                 subtree_data:add_le(buffer(databegin, datasize), info)
  122.         elseif cmd == 1023 then
  123.                 local info = "rcon login successful: "..buffer(databegin, datasize):string()
  124.                 subtree_data:add_le(buffer(databegin, datasize), info)
  125.         elseif cmd == 1024 then
  126.                 local info = "rcon not available: "..buffer(databegin, datasize):string()
  127.                 subtree_data:add_le(buffer(databegin, datasize), info)
  128.         elseif cmd == 1025 then
  129.                 local info = "rcon command: "..buffer(databegin, datasize):string()
  130.                 subtree_data:add_le(buffer(databegin, datasize), info)
  131.         elseif cmd == 1026 then
  132.                 local info = "rcon command failed: "..buffer(databegin, datasize):string()
  133.                 subtree_data:add_le(buffer(databegin, datasize), info)
  134.         elseif cmd == 1027 then
  135.                 local info = "rcon command successful: "..buffer(databegin, datasize):string()
  136.                 subtree_data:add_le(buffer(databegin, datasize), info)
  137.                
  138.         elseif cmd == 1005 then
  139.                 local info = "Use vehicle: "..buffer(databegin, datasize):string()
  140.                 subtree_data:add_le(buffer(databegin, datasize), info)
  141.         elseif cmd == 1008 then
  142.                 local oobtree = subtree_data:add(buffer(databegin+0,16), "OOB Data (16)")
  143.                 oobtree:add(buffer(databegin+0,4), "Time: "..buffer(databegin+0,4):le_uint())
  144.                 oobtree:add(buffer(databegin+4,4), "Engine Speed: "..buffer(databegin+4,4):le_uint())
  145.                 oobtree:add(buffer(databegin+8,4), "Engine Force: "..buffer(databegin+8,4):le_uint())
  146.                 local flagtree = oobtree:add(buffer(databegin+12,4), "Flags: "..buffer(databegin+12,4):le_uint())
  147.                 –flaghorn = buffer(12,4):le_uint() & 0×00000001
  148.                 –flagtree:add(buffer(12,4), "Horn: "..flaghorn)
  149.                 – todo: all all others
  150.                
  151.                 subtree_data:add(buffer(databegin+16,datasize-16), "Node Data ("..(datasize-16)..")")
  152.                
  153.                
  154.         else
  155.                 local info = "RoR Data ("..buffer:len()..")"..cmd
  156.                 local subtree = tree:add(ror_proto,buffer(),info)
  157.                 pinfo.cols.info:set(info)
  158.         end
  159. end
  160. – load the tcp.port table
  161. tcp_table = DissectorTable.get("tcp.port")
  162. – register our protocol to handle tcp port 12308
  163. tcp_table:add(12000,ror_proto)

i think you should be able to experiment with that now ;)
have fun! :)

Posted in coding | 1 Comment »

custom munin plugins the easy way (or: python rocks, perl sucks)

January 2nd, 2008

i just wrote this little neat python plugin for munin, to be able to track some lighttpd statistics

Python [Show Plain Code]:
  1. #!/bin/env python
  2. import sys
  3. if len(sys.argv) == 2 and sys.argv[1] == "autoconf":
  4.         print "yes"
  5. elif len(sys.argv) == 2 and sys.argv[1] == "config":
  6.         print ‘graph_title Lighttpd Stats’
  7.         print ‘graph_vlabel count’
  8.         print ‘graph_category lighttpd’
  9.  
  10.         print ‘total_accesses.label Total Accesses’
  11.         #print ‘Total_Accesses.draw AREA’
  12.         print ‘total_accesses.type COUNTER’
  13.  
  14.         print ‘total_kbytes.label Total kBytes transferred’
  15.         #print ‘Total_kBytes.draw AREA’
  16.         print ‘total_kbytes.type COUNTER’
  17.  
  18.         print ‘total_uptime.label Total Uptime’
  19.         #print ‘Total_Uptime.draw AREA’
  20.         #print ‘Total_Uptime.type GAUGE’
  21.  
  22.         print ‘total_busyservers.label Total Busy servers’
  23.         #print ‘Total_BusyServers.draw AREA’
  24.         #print ‘Total_BusyServers.type GAUGE’
  25.  
  26.         print ‘graph_args –base 1000′
  27. else:
  28.    import urllib2
  29.    f = urllib2.urlopen(‘http://127.0.0.1/server-status?auto’)
  30.    temp = f.read()
  31.    f.close()
  32.    temp = temp.replace(" ", "_")
  33.    temp = temp.replace(":_", ".value ")
  34.    print temp.lower()

EDIT: got an error in the first version, fixed that in the above one
EDIT2: now its finally working :)

Posted in server, python, coding, website | 2 Comments »

munin and custom database statistics

January 2nd, 2008

nice and easy to achieve, in just a few steps:

1. create the plugin

Below is an example for a membercount graph. You will have to replace the <> tags in the file or change the sql and/or legend.

/usr/libexec/munin/plugins/forum_members

#!/bin/sh
#%# family=auto
#%# capabilities=autoconf

#MYSQLOPTS="$mysqlopts"
MYSQL=${mysql:-mysql}

if [ "$1" = "autoconf" ]; then
        echo yes
fi

if [ "$1" = "config" ]; then
        echo 'graph_title Forum Members'
        echo 'graph_vlabel members'
        echo 'graph_category forum'
        echo 'total.label total members'
        echo 'graph_args --base 1000'
        exit 0
fi
A=$($MYSQL -u -p
 –disable-column-names –silent -e “SELE
CT count(ID_MEMBER) FROM forum.smf_members;”)

echo “total.value $A”

2. link to it:

cd /etc/munin/plugins
ln -s /usr/libexec/munin/plugins/forum_members

3. restart munin-node

/etc/init.de/munin-node restart

Posted in website | No Comments »

munin on amd64

January 2nd, 2008

how to install the server and the clients:

on the server and the clients:

echo "net-analyzer/munin ~amd64" >> /etc/portage/package.keywords
USE=-ssl emerge -vND munin

note1: i just had problems with the ssl version, also im using that in a secure environment, so i disabled it
note2: or make it persistent:

echo "net-analyzer/munin -ssl" >> /etc/portage/package.use

setup a client:

munin-node-configure --suggest --shell
# copy & paste commands on the screen
# remove those entries you dont want to be monitored (i.e. like nfsd)
# edit the configuration
vi /etc/munin/munin-node.conf
# and replace that: "allow ^127.0.0.1$" with your trusted (internal) IP
# and host* with your internal IP to listen on

# start the client:
/etc/init.d/munin-node start
rc-update add munin-node default

now we will setup the server:

vi /etc/munin/munin.conf

add entries like this example:

[mywebserver]
    address 192.168.1.40
    use_node_name yes

start it :) 

emerge --config net-analyzer/munin

if you want to directly view the graphs online, you could use lighttpd:

emerge -v lighttpd
/etc/init.d/lighttpd start

and browse to the server’s IP :)

EDIT:
if you have used the ssl USE - flag you need to create a certificate for the clients to get them working:

cd /etc/munin
openssl req -new -x509 -days 3650 -out munin-node.pem -keyout munin-node.pem -nodes

Please not that i had many problem with the ssl mode (i.e. clients will not end themself), so i would recommend you the non sll mode…

Posted in gentoo, website | 1 Comment »

current Xen infrastructure

January 2nd, 2008

this is the current infrastructure of this server (IPs replaced):

net.png

i will create some tutorial in order to help you to recreate that your own way :)

Posted in website | 1 Comment »

logical volume manager & backups

January 2nd, 2008

LVM is really cool to use, but two features are lacking in my opinion:
1. a nice backup feature
2. a volume merge feature (could be combined with the backup feature)

the best backup process to date possible which i found is that: http://tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html

Thats in short:
1. create a backup logical volume: 

lvcreate -L20G -s -n backup /dev/vg/webserver

(be sure that the size is more or equal the target volume size!

2. mount the backup volume:

mount /dev/vg/backup /mnt/

3. create a backup using the famous tar tool:

tar -cfvj /backup.tar.bz2 /mnt/

4. unmount the backup volume and remove it:

umount /mnt/ops/dbbackup
lvremove /dev/ops/dbbackup

not very comfortable.

EDIT: ah btw happy new year ;)

Posted in server, hardware | 1 Comment »

back to life

December 30th, 2007

after nearly 4 months and 2 server moves, this blog is back ;)
i hope i find the time to update it with some content.

Posted in website | 1 Comment »

tolua++ in combination with scons

August 20th, 2007

tolua is a program that can generate c/c++ interface code from simplified header files. so how to integrate this into your buildsystem?
here is a simple and elegant solution:

sconscript:

Python [Show Plain Code]:
  1. import os, glob
  2. Import("env")
  3. localenv = env.Copy()
  4. localenv.Append(CPPPATH = ["#/lua-5.1.2/src"])
  5. localenv.Append(LIBS = ["liblua", "dl","libtolua++"])
  6. localenv.Append(LIBPATH=["#/lua-5.1.2/src"])
  7. # find and add all .cpp files
  8. sources = glob.glob(‘*.cpp’)
  9. # this builds the interface
  10. lua_interface = ‘lua_interface.cpp’
  11. localenv.Command(lua_interface, ‘lua_functions.pkg’, "tolua++ -o $TARGET $SOURCE")
  12. # depend the build on the interface file
  13. sources.append(lua_interface)
  14. localenv.Program("test", sources)

in this example, “lua_functions.pkg” is the simplified header file for what the code should be generated.

Posted in scons | No Comments »

How to install QT4 under windows

May 26th, 2007

Just download it here:
http://trolltech.com/developer/downloads/qt/windows
be sure to use the mingw version!

after the download open a command line in the downloaded directory and start configure:
(be sure to use those options, because the default is to take the system libs)

configure -qt-zlib -qt-gif -qt-libpng -qt-libmng -qt-libjpeg

after configure finishes, you may start the build process:

mingw32-make

and then you have app. 60 minutes time to do other useful things (cooking dinner, having 1-3 cups of coffee ;)

Posted in windows, tutorials, coding | No Comments »

How to compile QT-win 3.3 under windows

May 26th, 2007

It is not such easy as you may think :)

After you downloaded and installed the Free Visual Studio 2005 c++ Express Edition, you must first follow this guide to install the Windows SDK:
http://msdn.microsoft.com/vstudio/express/visualc/usingpsdk/

Addintionally you must modify this file: C:\Program Files\Microsoft Visual Studio 8\Common7\Tools\vsvars32.bat and append all those include and lib path as in the howto above by MS. After this is done, you may follow this guide:
http://qtwin.sourceforge.net/qt3-win32/compile-msvc-2005.php

Please note that the original (and so latest) free QT version does not support msvc as compiler (For this you must buy QT), so i will use this custom mod of qt which re-enables msvc in QT 3.3.

Happy compiling !

Posted in windows, tutorials, coding | No Comments »

scons tutorial 1

May 25th, 2007

This first tutorial will try to create a basic working build system for platform-independent projects. These tutorials about SCons should help with common problems, and should start discussion about the best used techniques in common cases. For example we have the following directory structure:

dirstruct.png

Read the rest of this entry »

Posted in tutorials, scons, python, coding | 4 Comments »

use SCons!

May 25th, 2007