I’m coming to the conclusion that having code loop millions of times to copy data from one structure to another is never going to be fast.
I’m using two controls, a spreadsheet control and a report printing control. Unfortunately the two do not support the same data structures so to print a report Killink has to copy the data from the spreadsheet to the report which means looping. And when there are a million + rows with 3 columns the code is looping 3 million + times. Not much can be done to speed it up.
What I need is a way to keep the data in a single data structure that is supported by both components. I’m starting to have a better understanding why certain commercial software tends to not rely on 3rd party components and libraries. However, time is important to me so I must rely on 3rd party components for certain features.
For now Killink will have to warn the user before allowing a huge files to be printed. “You wish to print a file that is very large. It will take some time to prepare the data for printing. Do you wish to continue?”
The report I’m trying to preview during testing will have more than 15,000 pages. I don’t see anyone realistically printing this many pages but you never know.
The first version of Killink is feature complete, but there was one thing that was brothering me, large file support. Opening, printing, and saving a large file in Killink is a rather slow process so I’ve been working on improving these 3 areas.
The first area I worked on was opening a large file, which is what I will talk about today. After profiling the application I found the biggest bottleneck was populating the spreadsheet with display data. This was an easy fix that involved “freezing” display while the spreadsheet is loaded with data. This change resulted in a huge performance increase. The test file I used took minutes to load prior to the change and only 2 seconds after the change. But then Killink took another perform hit.
A beta tester reported the bug that embedded new line characters were not properly handled. I modified the CSV parser to check each byte of data as it was read from the hard drive. However this is a very inefficient approach. A better approach is to buffer each read. In other words, instead of reading a single byte from the hard drive it is better to read a chuck of bytes. The chuck can be any size such as 65K chucks or 1MB chucks.
Buffering the file read process gave Killink the performance boost it needed to open large files. It’s not as fast as I would like it but it’s getting closer.
The performance improved version of Killink should be ready for download in a couple of days.
CSV (also known as comma separated values, comma separated list, or comma separated variable) is a commonly used file format that stores tabular data. Each data element, or value, is separated with a comma or other single character delimiter such as a Tab character. These types of files are also called delimited text files or flat files.
Many software application support CSV delimited text files either as an import or export data format. For example, Outlook users can export the address book to a comma separated value file and import the contacts into another application.
I have worked with comma and tab delimited text files throughout most of my career. I have written applications that produce delimited text and consume delimited text. And while XML is my preferred file format for sharing data there is no escaping the usefulness of CSV delimited text files.
While CSV delimited text files are typically used to import and export data between software applications there are times when individuals must manually create or edit comma and tab delimited text files. And many of these individuals use a spreadsheet program such as Microsoft Excel. However, using a full featured spreadsheet such as Excel to create and edit CSV delimited text files seems overkill to me.
For starters, say you only need to manipulate CSV delimited text files and have no need for working with spreadsheets. The licensing cost of Excel is hardly justified. Secondly Excel imposes certain size limitations. Excel is limited to 65,536 rows of data and 256 columns per worksheet. This means you will not be able to open a delimited text file containing 100,000 rows. Faced with high licensing cost, overkill list of features, and size limitations I decided to write Killink.
Killink is a spreadsheet-like editor for delimited text files. It supports delimited text files using any single character delimiter including comma and tab, and it supports large delimited text files.
Version 1.0 is currently in beta test which is open to the public. Visit the Killink download page and give the latest release a test drive. I appreciate any feedback, good or bad, you might have about Killink. Feedback can be posted to the Killink Support forum.
Update: Microsoft has improved the limitation of Excel. Excel 2007 supports 1 million rows per worksheet.
Writing code while listening to Led Zeppelin puts me into the zone especially when I’m making bug fixes.
I’m adding a new feature to Killink to allow the user to integrate the application with Windows Explorer. Not a difficult task. Just involves making some registry settings. But one registry setting I need to make must go under HKEY_CLASSES_ROOT, and this hive is under administrative control in Vista.
This isn’t such a big deal if I am using an installer. Installers will typically ask for elevated permissions to perform the install. But I’m trying to provide a version that does not require an installer. This means my application will have to ask for elevated permissions when the user clicks the option to integrate with Explorer. It also means I need to display the shield icon indicating that elevated permissions are required to perform the task.
This really has me bummed out because it means more work (and learning). Now I have to learn up to ask for elevated permissions within the application.
Sigh…
Jim Douglas replaces Ben Smith as CEO for CodeGear. This move makes no sense to me, but then again I have never really understood Corporate America.
FinalBuilder should have support for Delphi 2007 next week. Sweeeet.
The 7th thing I like about Delphi 2007 is the “non-breaking” release. In a nutshell this means DCU and packages compiled for BDS 2006 will work with Delphi 2007. I initially had problems with some of the 3rd party packages I manually installed in Delphi 2007 so I didn’t list this as one of the 5 things I like about Delphi 2007. But after resolving the issues (caused by me) I’m finding NO problems using 3rd party packages built for BDS 2006 in Delphi 2007.
<ul>
<li>Things I Like About Delphi 2007 #1-5</li>
<li>Things I Like About Delphi 2007 #6</li>
</ul>
The number 6 thing I like about Delphi 2007 is the new message output for compiles. It provides detailed information about the compile including the dcc32.exe command line. It’s nice to see this level of detail especially when odd things start to happen.
In my posting 5 Things I Like About Delphi 2007 I mentioned debugging problems I experienced after installing 3rd party packages. The source of the problem was in fact the use of certain 3rd party packages, more specifically the DevExpress VCL packages. However the problem wasn’t with the DevExpress directly.
I installed all packages found in the DevExpress library/delphi10 directory. This was a mistake made by me. Only certain design time packages need to be installed in the IDE, not all packages in that directory. Lucky for me I still have BDS 2006 installed.
I launch BDS 2006 and Delphi 2007. I went to the installed package list in BDS 2006 (Component > Install Packages…) and manually added each DevExpress package listed in BDS 2006 to Delphi 2007. This resolved the debugging issues I experienced yesterday. And now I’m able to make use of Delphi 2007 for existing projects.