C++ :: Parsing Large Text File
Feb 10, 2013
I have a massive text file containing many thousands of directory and file names with / at the root, like so:
/dir/
/dir/dir/
/dir/dir/dir/
/file
/dir/file
/dir/dir/file
/dir/dir/dir/file
I need to parse the file in such a way that I can create a filesystem hierarchy as if I were enumerating files/directories. Ultimately I want to add these to a tree gui control with everything under its proper node without duplicating anything. It should look roughly like so:
dir
-file
-dir
-file
-dir
-file
I can open the file and add nodes/children to the tree control but how should I go about doing the actual parsing? How can I find a filename and say "this belongs under this node"? I want to do this efficient as possible even if I must use multiple threads.
View 1 Replies
ADVERTISEMENT
Dec 16, 2013
I have a large binary file (84GB) that needs to be broken down into smaller file sizes (~1GB to 8GB) for analysis. The binary file is on a 32-bit machine and cannot be copied to another machine for analysis. The PC has Visual Studio 6.0 and is not upgradable. My issue is I'm using the following generic code to create the smaller files.
fseek(file, start, SEEK_SET);
end = start + (variable based on file size);
fseek(file, end, SEEK_SET);
for (i=start; i<end; i++) {
if(!feof(f)) {
byte = fgetc(f);
fputc(byte,new_file);
}
}
However, on a 32-bit machine, the iterator can only count up to ~2billion. Which means that I'm unable to copy anything past ~2GB. My original idea was to delete from the large binary file as I read from it so that I can reset the iterator on every read. However, I haven't come across a way to delete binary file entries.
Is there any other way that to break down a large binary file into smaller units? Or is there a way to delete binary file entries in sections or per entry?
On a 64-bit machine I could use _fseeki64. I've been reading that some versions of Visual 6.0 are capable of supporting 64-bit numbers but when using _fseeki64 or _lseeki64 on this machine its an "undeclared identifier"
View 7 Replies
View Related
May 30, 2013
So I'm attempting to write a program that will parse through a large file (genome sequences) and I'm basically wondering what options I should consider if I wanted to either:
a) store the entire genome in memory and then parse through it
b) parse through a file in small portions
If I go with "a", should I just read a file into a vector and then parse through it? And if I go with "b" would I just use an input/output stream?
View 5 Replies
View Related
Aug 26, 2014
Requirements in filtering the text file.
1. first my professor required me NOT to change the MAIN function(because he made it)
2. I have to make 3 getlogs() STRING FUNCTIONS:
a. string getlogs(); - accepts no paramters, SHOWS ALL THE CONTENTS OF TEXT FILE
b. string getLogs(const string & a); - accepts 1 parameter -SHOWS ONLY THE LINE WHICH CONTAINS THE SPECIFIED DATE FROM MAIN FUNCTION which is "2014-08-01"
c. string getLogs(const string & b, const string & c); - accepts 2 parameters, SHOWS ONLY THE LINES FROM THE DATE START to DATE END specified at THE MAIN FUNCTION which is date start-"2014-08-01";DateEnd = "2014-08-10";
3. all COUT should be in the MAIN FUNCTION
TEXTFILE CONTAINS:
2014-08-01 06:13:14,Name,4.5,CustomUnit,CustomType
2014-08-02 06:13:14,Name,4.5,CustomUnit,CustomType
2014-08-03 06:13:14,Name,4.5,CustomUnit,CustomType
2014-08-04 06:13:14,Name,4.5,CustomUnit,CustomType
[Code] .....
my codes so far:
//MAIN
#include <iostream>
#include <string>
#include <fstream>
#include <dirent.h>
[Code].....
View 1 Replies
View Related
Dec 8, 2014
I'm parsing a text file, and I'd like to detect when a certain Compilation Condition - i.e. #ifdef - begins. The challenge is, that the condition can take any of the following patterns:
#ifdef (FLAG)
#if defined (FLAG)
#if (defined (FLAG))
(And perhaps I missed more)
I'd of course need to treat them all the same, as they are indeed the same. How would you know to treat them all the same?
View 2 Replies
View Related
Jan 1, 2014
I am currently working out on a problem in which a c program is to be made which shows a large text file in parts.
f
For example: If file contains 200 lines. 50 lines will be shown on first page and user is asked to press any key to move to next page until EOF is found. user is allowed to return to previous page as well, and this is very complicated task for me. I tried to move cursor to a specific position using fseek etc but it page doesn't stop and reaches to end quickly.
View 1 Replies
View Related
Nov 7, 2014
I am writing a spell checker for a exercise for a class I am taking online. I have to load a huge text file of 143092 words into a hash table.
The program segfaults on word number 63197. Is there a way to debug this in gdb without having to step threw the function 63197 times.
View 3 Replies
View Related
Mar 31, 2014
I am creating a simple log parser (loads a text file and filters out unnecessary information, but has the option to show the full log) and I'm running into an issue with fairly large log sizes (50+mgs). I have seen a few recommendations from a stream to memory manged files and even alternate 3rd party controls.
I foresee a few issues with any of the non-third party solutions (which I would prefer to avoid third-party add-ins) such as the scroll bar not correctly reporting the relative length or position of the complete text in the box (when displaying only a portion of the file at a time) and in the stream solution where you read on scroll (as necessary) have not only the same issues, but how do you resume reading in the middle of the file? This also all assumes I would be periodically clearing the RichTextBox to keep the memory usage down to avoid an OutOfMemoryException (which I have been running into.)
View 14 Replies
View Related
Apr 11, 2014
I'm parsing an xml file full of payslips and using the data in another application. I've got it all working but I suspect it isn't the most elegant piece of code. I run through the xml file finding a series of "Text" attributes/elements" and then I run through it again finding a series of "Field" attributes/elements, Here is a sample of the code:
For getting the "Text" fields :
XNamespace ns = "urn:crystal-reports:schemas:report-detail";
//
// Get all the Text attributes & Elements
//
foreach (XElement xtxt in xdoc.Descendants(ns + "Text"))
[Code]...
This works fine, I extract all the data I'm interested in and go to do my thing with it. However I really need to know when each record ends and I was doing that by looking for "Text24" in the text fields and "EeRef2" in the field fields, which wasn't very elegant in the first place. Then a "Text16" was added to end of each record which was fine I could just look for "Text16" but now it's apparent that "Text16" isn't always there. I've got it all working for now but I'd prefer to process one record at a time i.e. extract all the "Text" & "Field" values for one record, do whatever I need to do with it, update the xml file to indicate this progress ( if possible ) and then move on to the next record. I've attached a sample of the xml but basically is has the following structure :
<Details>
<Section>
<Text></Text>
"
<Field></Field>
[Code]...
So a record is everything between the first <Details> and the last </Details> with two <Details> and two </Details> in between.
View 2 Replies
View Related
Mar 26, 2015
I have this file that I would like to read into a multidimenstional array in c#: file. If I take the first set of lines as a first example, I would like the print_r to look something like this:
Array (
[SiiNunit] => Array (
[0] => Array (
[nameless.2BB8.4FB8] => Array
[0] => Array (
[bank] = _nameless.2917.43B0
[player] = _nameless.2813.6928
[companies] = 312
[code].....
this (as I expected it to) wrote each line of the file into a 1d array however, I would like it to write it to a multidimensional array.
View 7 Replies
View Related
Dec 2, 2013
Are there any examples how to parse a matrix:
Code:
const string ABC = "
A B C D
A 1 -1 2 14
B 0 -2 -4 8
C 6 2 2 3
" so if i have it as a string stream and then loop through each line like this:
Code:
istringstream in (ABC);
for (string line; getline(in, line); ){
vector<char> vec(line.begin(), line.end());
for (int i = 0; i< vec.size(); i++)
cout << vec[i] << "
";
}
I get my strings chopped into characters. but how to chop it into "meaningful" characters so that -1 is not - and 1. is there any quick way for that to happen ??
View 3 Replies
View Related
Sep 20, 2013
I have been attempting to store mathematical functions in a file by parsing them into a linked list with a variable sized char ** array as my storage device. I have ran into problems with the memory management detail. The program crashes before output is flushed to the console, so printf() wasn't a debugging option. Neither is my actual debugger, since it seems to get a SIGTRAP every time. I have my warnings turned all the way up, but no errors or warnings are appearing. The part I know works is the actual code that opens the file and gets a line from the file. As far as the two functions that implement the linked list, that is most likely where the problem lies. My current attempt is basically to store the size of the dynamic array in the structure and keep resizing it until there are no more tokens. Then I will store the number of elements of the array in the structure and move on to the next node.
Here is my text file I use :
Code:
sqrt( 25 ); pow( 6 ); sin( 2 );
pow( 4 ); tan( pow( 2 ) ); Main.c :
Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct file_data
[Code]...
View 7 Replies
View Related
Jan 4, 2014
I am parsing a binary data file by casting a buffer to a struct. It seems to work really well apart from this one double which is always being accessed two bytes off, despite being in the correct place.
Code:
typedef struct InvoiceRow {
uint INVOICE_NUMBER;
...
double GROSS;
...
char VAT_NUMBER[31];
} InvoiceRow;
If I attempt to print GROSS using printf("%f", row->GROSS) I get 0.0000. However, if I change the type of GROSS to char[8] and then use the following code, I am presented with the correct number...
Code:
typedef struct example {
double d;
}
example;
example *blah = (example*)row->GROSS;
printf("%f", blah->d);
View 7 Replies
View Related
Nov 5, 2014
I am searching an API that can give me thumbnail(with large Icons) of all folders of all drive in Windows 7. Like
c:/--->Folder1,Folder2.....FolderN
d:/--->Folder1,Folder2.....FolderN
.
.
n:/--->Folder1,Folder2.....FolderN
How i can get associate thumbnail(with large Icons) of each folders of any of the drive.
View 5 Replies
View Related
Feb 21, 2015
I have a specific byte (that is unsigned char) array, that I need to find in a very big file (2GB or so), currently I'm using:
size_t fsFind(FILE* device, byte* data, size_t size) {
int SIZE = (1024 > size) ? 1024 : size;
byte buffer[SIZE];
int pos = 0;
int loc = ftell(device);
[Code] ....
Which seems to find proper result on first use, but on subsequent searches it seems to find invalid positions, is there something I'm doing wrong, or is there a better way?
View 6 Replies
View Related
Dec 10, 2013
I need a large string vector for file content manipulation.
v1.max_size() gives me 153 391 689.
But if I make a test by looping a vector pushing back strings it crashes around 16 - 26 000 000 though I still have a lot of ram (1GB left).
How come and why isn't vector size limited by ram instead?
View 11 Replies
View Related
Apr 1, 2015
Im trying to read and store several students information so that i can have an options menu where i can enter a student number and the program prints all the information stored about that student. The way i have it set up now, doesn't work for this because all info is reinitialized to stud1. Is there another way to store this info other than defining stud1, stud2,.....,stud200?
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
struct student_info {
char first[20];
char last[20];
[Code]....
View 1 Replies
View Related
Nov 29, 2014
Code software that, from an original text file, generate another file with the text content in upper case.For exemple:
entrence:
luke
tom
alex
outings:
LUKE
TOM
ALEX
My code so far:
Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
}
[code]....
View 2 Replies
View Related
Jun 5, 2014
I want to make a program that opens a text file and checks the usernames listed in the text files to see if the names are registered on a site such as twitter. How easy would this be to make, what things would I need to know?
View 4 Replies
View Related
May 31, 2013
I am trying to get text file and read only first 100 lines of the text content using c/c++. how can i do it?
is it something like string line;
getline(stream,line);
??
View 7 Replies
View Related
Aug 30, 2013
I have a huge text file in following format:
{Bunch of text
.
.
}
[Code].....
I want to extract Text1, Text2, Text3, Text4,..., Text600 in the output file. How can i achieve this?
/* BTW, I am not getting my homework done here. I am an ex-programmer, who has now moved to marketing for some time now, and today, I encountered this problem, which I believe can be solved easily through programming. */
View 3 Replies
View Related
Aug 31, 2014
i want to create 100 gmail accounts instantaneously....what i want from you guys is i have written a program that create a text file i want that once i give the program the imput of 1 it should delete the first 3 lines from the file i.e. the first account details coz that is already been created and shift the rest of it 3 lines upwards after that i'll write a javascript that will automatically fill and create the accounts with those names in web browser.....my lil program is here:
#‎include <stdio.h>
#include <conio.h>
main()
[Code]....
View 1 Replies
View Related
Aug 2, 2014
I have a text file called (Test.txt) with the following text:
This is line one
This is line two
This is line three
How do I go about writing a program that will print a line of text (e.g. "This in line two" on the console screen?
All I can seem to do is display the entire text file.
View 6 Replies
View Related
Oct 1, 2012
So im trying to parse a string into a Ip Address but i have a problem, the IPAddress.Parse method only works for ipv4 address's how do i parse ANY Ip address into a string, if i use the IPaddress.Parse method on my public(remote) IP it throws an exception but on ipv4 local ip it doesn't how do i parse ANY ip address the user inputs as a string as an Ip Address?
View 5 Replies
View Related
Sep 10, 2014
I have been here for almost 3 months looking for answers in my C++ problems.here's some type of code for this.
cout << "Enter value of x: " << endl; //Let's say 5.
cin >> x;
cout << "Enter equation: "; //Let's say x+1
cin >> equation;
Then the program analyzes that this character "x" has an initial value of 5.I already have the parser for the equation functions (+,-,*,/)This is the only thing lacking. Is there some type of function that i missed?
View 6 Replies
View Related
Nov 3, 2013
I have an input file that contains any number of lines. Each line will follow the same structure. There will be 5 fields, separated by 4 commas. Fields 1 and 3 will be single characters, fields 2,4,5 will be integers. Example:
<A, 123, B, 456, 789>
I need to iterate over each line, parse out the fields, and capture each field value into a separate variables.
header file: Code: #include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/socket.h>
[Code]....
I've used similar code to this before, but this time I'm getting the dreaded pointer from integer error on my strtok lines:
warning: passing argument 2 of 'strtok' makes pointer from integer without a cast'
View 2 Replies
View Related
