Hack Week: Day 5 (Friday) – The last day

Kohei Yoshida — Sat, 30 Jun 2007 05:26:51 +0000

Well, today was the last day of Hack Week, but unfortunately I wasn’t able to squish the remaining 20% unconverted resource files like I planned to do yesterday. I squished only 5%. This brings my conversion success rate from yesterday’s 80% to 85%. I’m pretty happy with this result, however, considering that some of those resource files I tried to convert are not even dialog resource files.

Here is what I did today:

Fixed incorrect expansion of preprocessing macros. It just didn’t do the right thing when performing recursive macro expansion. This time I really got it right, but it consumed the majority of today’s hacking time. :-(
Reworked my expression evaluation code to fully support the reverse Polish notation (RPN). The absence of this feature caused a parse failure on some files because the position and the size of some widgets are given as a mathematical expression (e.g. (24 + 10)/2) instead of a single number. I got the RPN parser to work, but then I realized that I could have just used Python’s builtin eval function to evaluate a whole expression in one step. Well, duh! I learned how to code the RPN builder to evaluate an expression, though, which was fun exercise.

So, this concludes this week’s Novell Hack Week event. It was certainly fun, although I couldn’t do everything I wanted to do. I’ll be back to my normal hacking activities on next Monday.

Hack Week: Day 4 (Thursday) – The joy of preprocessing macros (not!)

Kohei Yoshida — Fri, 29 Jun 2007 14:25:49 +0000

Well, I didn’t have much huge achievement yesterday – day 4 of our Novell’s Hack Week. But here is a list of things I’ve done to improve the robustness of the converter script.

Added support to (semi-)correctly parse the preprocessing macros, both ones that take no arguments and ones that do take arguments, as well as ones that include other macros recursively.
Added support to parse header files, without which many preprocessing macros would be left undefined, thus causing a parse failure.
Added arithmetic support, again in the preprocessing macros.
Numerous bug fixes that were uncovered while working on the preprocessing macro support, as well as some re-write of the algorithms to make them work better.

My conclusion? Preprocessing macros are evil! Since macros are expanded before the source file is parsed, it has its own syntax rules that are different from the host language. A simple expansion is rather easy, but once they start taking arguments, recursively using other macros (or the combination of the two), things become a bit tricky. Anyway, the worst is over I hope…

With this improvement, I can now correctly convert 80% of all of the src files we have in our OO.o source tree. Hopefully I can squish the remaining 20% today.

To recap (for those who missed my previous Hack Week posts), I am working on writing a .src to .xml converter script to migrate the existing dialog resource files (which are statically designed) to new xml format that has layout information. The new xml files will be used as a starting point for re-designing all our existing dialogs for the new dialog layout engine in development.

Hack Week: .src converter to convert ~700 .src files

Kohei Yoshida — Thu, 28 Jun 2007 05:21:03 +0000

So, after some discussion with Ricardo, I have decided to take on the task of writing a converter script to convert ~700 .src files into xml files, which will be used as a starting point for re-designing each and every dialog for the new dialog layout engine. I was initially thinking about working on the dialog editor, but sounds like Ricardo has it under control. So better not mess with that. :-)

When writing a converter script, it of course involves parsing a source file in order to generate output. Typically there are two ways to go about this.

Parse the source file partially for just the information you need using a flat search, and ignore the rest, or
Parse the source file fully according to the syntax of the language, using a lexer-parser pattern.

The advantage of the first method is simplicity; it’s pretty easy to set up a simple regexp-based parser and start parsing. The disadvantage of it is that, once the parsing need grows, as you need to pick up more and more information, the parser code becomes complex with full of special case handling, and eventually requires a total re-write. Good luck with extending such code as the need grows even further.

The second method, while it takes a little upfront effort, is extensible once the framework is set up, and the code usually becomes better structured with only a minimum special case handling if designed correctly. This method is also well-suited for parsing a token-based language, where whitespace and linebreak characters are only for syntactic sugar and does not affect its semantics. For example, C/C++ and Java are token-based, while Python is not. Since the syntax of the src files is very similar to that of C, I’ve decided to use the second method for this task.

I spent yesterday and today writing this converter script from scratch (in Python), and I’ve come to a point where it parses a large number of src files and correctly generate their xml output files. Here is one example case.

The source file:

/*************************************************************************
 *
 *  OpenOffice.org - a multi-platform office productivity suite
 *
 *  $RCSfile: crnrdlg.src,v $
 *
 *  $Revision: 1.44 $
 *
 *  last change: $Author: ihi $ $Date: 2007/04/19 16:36:48 $
 *
 *  The Contents of this file are made available subject to
 *  the terms of GNU Lesser General Public License Version 2.1.
 *
 *
 *    GNU Lesser General Public License Version 2.1
 *    =============================================
 *    Copyright 2005 by Sun Microsystems, Inc.
 *    901 San Antonio Road, Palo Alto, CA 94303, USA
 *
 *    This library is free software; you can redistribute it and/or
 *    modify it under the terms of the GNU Lesser General Public
 *    License version 2.1, as published by the Free Software Foundation.
 *
 *    This library is distributed in the hope that it will be useful,
 *    but WITHOUT ANY WARRANTY; without even the implied warranty of
 *    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 *    Lesser General Public License for more details.
 *
 *    You should have received a copy of the GNU Lesser General Public
 *    License along with this library; if not, write to the Free Software
 *    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
 *    MA  02111-1307  USA
 *
 ************************************************************************/
#include "crnrdlg.hrc"
ModelessDialog RID_SCDLG_COLROWNAMERANGES
{
    OutputSize = TRUE ;
    Hide = TRUE ;
    SVLook = TRUE ;
    Size = MAP_APPFONT ( 256 , 181 ) ;
    HelpId = HID_COLROWNAMERANGES ;
    Moveable = TRUE ;
     // Closeable = TRUE;   // Dieser Dialog hat einen Cancel-Button !
    FixedLine FL_ASSIGN
    {
        Pos = MAP_APPFONT ( 6 , 3 ) ;
        Size = MAP_APPFONT ( 188 , 8 ) ;
        Text [ en-US ] = "Range" ;
    };
    ListBox LB_RANGE
    {
        Pos = MAP_APPFONT ( 12 , 14 ) ;
        Size = MAP_APPFONT ( 179 , 85 ) ;
        TabStop = TRUE ;
        VScroll = TRUE ;
        Border = TRUE ;
    };
    Edit ED_AREA
    {
        Border = TRUE ;
        Pos = MAP_APPFONT ( 12 , 105 ) ;
        Size = MAP_APPFONT ( 165 , 12 ) ;
        TabStop = TRUE ;
    };
    ImageButton RB_AREA
    {
        Pos = MAP_APPFONT ( 179 , 104 ) ;
        Size = MAP_APPFONT ( 13 , 15 ) ;
        TabStop = FALSE ;
        QuickHelpText [ en-US ] = "Shrink" ;
    };
    RadioButton BTN_COLHEAD
    {
        Pos = MAP_APPFONT ( 20 , 121 ) ;
        Size = MAP_APPFONT ( 171 , 10 ) ;
        TabStop = TRUE ;
        Text [ en-US ] = "Contains ~column labels" ;
    };
    RadioButton BTN_ROWHEAD
    {
        Pos = MAP_APPFONT ( 20 , 135 ) ;
        Size = MAP_APPFONT ( 171 , 10 ) ;
        TabStop = TRUE ;
        Text [ en-US ] = "Contains ~row labels" ;
    };
    FixedText FT_DATA_LABEL
    {
        Pos = MAP_APPFONT ( 12 , 151 ) ;
        Size = MAP_APPFONT ( 179 , 8 ) ;
        Text [ en-US ] = "For ~data range" ;
    };
    Edit ED_DATA
    {
        Border = TRUE ;
        Pos = MAP_APPFONT ( 12 , 162 ) ;
        Size = MAP_APPFONT ( 165 , 12 ) ;
        TabStop = TRUE ;
    };
    ImageButton RB_DATA
    {
        Pos = MAP_APPFONT ( 179 , 161 ) ;
        Size = MAP_APPFONT ( 13 , 15 ) ;
        TabStop = FALSE ;
        QuickHelpText [ en-US ] = "Shrink" ;
    };
    OKButton BTN_OK
    {
        Pos = MAP_APPFONT ( 200 , 6 ) ;
        Size = MAP_APPFONT ( 50 , 14 ) ;
        TabStop = TRUE ;
    };
    CancelButton BTN_CANCEL
    {
        Pos = MAP_APPFONT ( 200 , 23 ) ;
        Size = MAP_APPFONT ( 50 , 14 ) ;
        TabStop = TRUE ;
    };
    PushButton BTN_ADD
    {
        Pos = MAP_APPFONT ( 200 , 104 ) ;
        Size = MAP_APPFONT ( 50 , 14 ) ;
        Text [ en-US ] = "~Add" ;
        TabStop = TRUE ;
        DefButton = TRUE ;
    };
    PushButton BTN_REMOVE
    {
        Pos = MAP_APPFONT ( 200 , 122 ) ;
        Size = MAP_APPFONT ( 50 , 14 ) ;
        Text [ en-US ] = "~Delete" ;
        TabStop = TRUE ;
    };
    HelpButton BTN_HELP
    {
        Pos = MAP_APPFONT ( 200 , 43 ) ;
        Size = MAP_APPFONT ( 50 , 14 ) ;
        TabStop = TRUE ;
    };
    Text [ en-US ] = "Define Label Range" ;
};

and here is the output after the conversion:

 height="181" help-id="HID_COLROWNAMERANGES" hide="true" moveable="true" output-size="true" sv-look="true" text="Define Label Range" width="256" xmlns="http://openoffice.org/2007/layout" xmlns:cnt="http://openoffice.org/2007/layout/container">
    >
         id="FL_ASSIGN" height="8" text="Range" width="188" x="6" y="3"/>
         id="BTN_OK" height="14" tab-stop="true" width="50" x="200" y="6"/>
         id="LB_RANGE" border="true" height="85" tab-stop="true" vscroll="true" width="179" x="12" y="14"/>
         id="BTN_CANCEL" height="14" tab-stop="true" width="50" x="200" y="23"/>
         id="BTN_HELP" height="14" tab-stop="true" width="50" x="200" y="43"/>
        >
             id="RB_AREA" height="15" quick-help-text="Shrink" tab-stop="false" width="13" x="179" y="104"/>
             id="BTN_ADD" def-button="true" height="14" tab-stop="true" text="~Add" width="50" x="200" y="104"/>
        >
         id="ED_AREA" border="true" height="12" tab-stop="true" width="165" x="12" y="105"/>
         id="BTN_COLHEAD" height="10" tab-stop="true" text="Contains ~column labels" width="171" x="20" y="121"/>
         id="BTN_REMOVE" height="14" tab-stop="true" text="~Delete" width="50" x="200" y="122"/>
         id="BTN_ROWHEAD" height="10" tab-stop="true" text="Contains ~row labels" width="171" x="20" y="135"/>
         id="FT_DATA_LABEL" height="8" text="For ~data range" width="179" x="12" y="151"/>
         id="RB_DATA" height="15" quick-help-text="Shrink" tab-stop="false" width="13" x="179" y="161"/>
         id="ED_DATA" border="true" height="12" tab-stop="true" width="165" x="12" y="162"/>
    >
>

These are the steps I take to convert each file. First, the source file is read character-by-character to get tokenized by the lexer class, and this is where the comments (both multi-line and single line) get stripped out and the preprocessing macros are defined. The tokens are then passed to the parser class to build a syntax tree (preprocessor macros are expanded here), which is then converted into an intermediate XML tree with names translated and some attribute types converted properly, such as the position and the size, which are originally given as MAP_APPFONT( a, b ) format. Also, some unnecessary information is discarded at this stage.

Once that’s done, it further translates the intermediate XML tree into another XML tree that has layout elements. The X and Y positions of each widget are used in order to layout the widgets properly by wrapping them with and elements as needed. The tree is then dumped into a stream of text, which is what you see above.

Unfortunately this task is not done yet. As it turns out, some src files even require inclusion of header files in order to be parsed correctly, which means I need to honor those #include "foo.hrc" header include directives. Right now, they are ignored. On top of that, there may also be cases where the #ifdef directives might need to be interpreted correctly, but so far ignoring them has not caused any side-effect.

I’m sure there are other problems I’ll encounter as I parse more src files, but I’d say the end is near. :-)

Roundtrip to Shanghai via Tokyo » src2xml

Hack Week: Day 5 (Friday) – The last day

Hack Week: Day 4 (Thursday) – The joy of preprocessing macros (not!)

Hack Week: .src converter to convert ~700 .src files