C++ :: Processing Large Files - Getting A Specific Line
Jul 5, 2012
I am reading in some large files to process. The files need to be parsed into multi-line sub units for processing (by a different physical process). My current setup is crude and parses the entire file into memory before beginning to process the sub units. This is fine as long as the file fits, so I get into trouble > 2GB or so when my machine flat runs out of memory. The simple thing to do is to only read in some of the file, process what was read in, and then read in more. I more or less know how to stop reading at some point in the file, but I'm not sure how to resume from that point later when I need more data. Is there a way to count lines and then getline() starting from a specific point in the file? From reading the getline() doc, it doesn't look like there is.
This is my current function,
Code:
void ParseFile(char *path, char type) {
// open input file
ifstream input_file(path);
if( !input_file.good() ) { cerr << "Failed to open " << path << endl; exit(ERRCODE_ERROR); }
int pos = 1;
[Code] ....
I need to remove the code to open the file from this function and open it elsewhere. I would like to call this every time I run out of data, if( tasks.empty() ), and get more data from the input file, but I don't know how to resume reading the input file from where I left off.
If this is not possible, I guess I would have to place a function call in while( input_file.good() ) to call out and process the data I have. When the current tasks list is finished, the list could be cleared and control returned into the while loop to continue reading input and start re-populating the list.
I've tried to form together the following code so that I can process multiple files from one user specified directory. However, when I run the program it works great for the first file, but all other files are blank. The problem starts on the following line:
I have a problem to read a large number of binary files, process them and store them under a new name. The program and routines go very well for 505 files. After reading 506 files, the program now refuses to read the next file. I have 16 Gb of memory and tried to close all other programs and restart the PC. it always stops after 506 files (512 files would be more understanding in a way...).
Here is my code. I have tried many things without success. This is only part of the loop that stops. The if test if (myfile.is_open() returns false by some reason. I can start the process again starting with the file that does not open and then it stops again after 506 files.
char * tfiBlock; ifstream myfile (OrigFilename, ios::in|ios::binary|ios::ate); if (myfile.is_open()) { int lengde = myfile.tellg(); tfiBlock = new char [lengde]; //static char memblock [size];
[Code] .....
Clean up procedure: delete[] tfiBlock;
Are there any limits to how many files that can be opened, or is it maybe someting to be set in the compiler?
I have managed to make a program that permutates a string with repetition.
I ran it to permutate "abcdefghijklmnopqrstuvwxyz1234567890" with a limit of 5 characters.
This took a little over 5 hours for my pc to process this and I ended up with a .txt 403MB in size. Needless to say I am unable to open this .txt in notepad without Notepad.exe not responding and me having to end the process.
So what I want to do is modify my code to break up the output in to several files rather than one. Possibly all permutations starting with a in one file, b in another, etc.
Here is my current code: #include <iostream> #include <string> #include <sstream>
[Code]....
As you can see it currently appends permutation.txt with all output. I would like it to make files like this permut_5char_a.txt, permut_5char_b.txt, etc.
I want to read a folder that contains large number of tif files. (say 1000 tif images) requirement is to read all files and store in a array/array pointer.
I am creating a simple log parser (loads a text file and filters out unnecessary information, but has the option to show the full log) and I'm running into an issue with fairly large log sizes (50+mgs). I have seen a few recommendations from a stream to memory manged files and even alternate 3rd party controls.
I foresee a few issues with any of the non-third party solutions (which I would prefer to avoid third-party add-ins) such as the scroll bar not correctly reporting the relative length or position of the complete text in the box (when displaying only a portion of the file at a time) and in the stream solution where you read on scroll (as necessary) have not only the same issues, but how do you resume reading in the middle of the file? This also all assumes I would be periodically clearing the RichTextBox to keep the memory usage down to avoid an OutOfMemoryException (which I have been running into.)
We have a program that sequentially processes a large number of files (currently about 700 expected to increase to about 1500). The program performs the same processing on each file (and doesn't involve any other file) which is io-bound and not cpu-bound. This process takes several hours and it is normally performed overnight.
I've refactored the program so that the processing for each file is done within its own thread (ie one thread created for the processing of one file). This gives rise to many hundreds of io-bound threads. This refactored program is working with no errors reported and has reduced the total processing time down to about 10 minutes.
Any problems that might arise having this number of threads (700 to 1500) created/running?
I'm trying to make sure my code is written in smaller modules, so my first step is to create my initialization process in and external file to load the necessary data from external sources and set up things like the content of drop down list boxes.
My first attempt failed to give me access to the combobox items add function so I moved that code back into the form1.h file:
Code: public: Form1(void) { InitializeComponent(); // //TODO: Add the constructor code here // } void AddDate(char *date, int ID) { this->comboBox1->Items->Add("line 1"); }
It compiles fine, but the call to it in my Initialize.cpp file
Code: MarketView::Form1::AddDate("abs",1); Gives error C2352: 'MarketView::Form1::AddDate' : illegal call of non-static member function
OK, so I change "void AddDate" to "static void AddDate" and now get the error that "static member functions do not have 'this' pointers" so I go back to the "MarketView::Form1::comboBox1" situation where there is no legal syntax after "Box1 to get me to Items->Add
I've been an old fashion programmer for over 47 years. It seems as is the concept of programming computers has changed from the concepts of logic to memorization of complex syntax.
There has to be a simple answer to do this other than to write thousands of lines of code in one Form1.h file. I refuse to believe that the new programming concepts will not allow you to write code in smaller more manageable modules.
What is the proper syntax for breaking up the larger file into more manageable chucks?
I am currently working on a small project, which is an ATM simulator but, lately, I've been stuck in a part where I have to overwrite a specific line in a .dat file to update the account's checking and savings balance, once the transaction has been done. I have to use StreamReader and StreamWriter to complete the task and I have also being told to try Collections... which I'm not sure as to why.
Here's my code so far:
private void btnOKd_Click(object sender, EventArgs e) { int transaction = 0; atmTIMER.Elapsed += new ElapsedEventHandler(timerRESTART); FileStream fileTRANSACTIONS = new FileStream("C:\Users\Voyage\Desktop\ATMSim\ATMsim\Transactions.dat", FileMode.Open);
I am suppose to make a program that will output a line for a specific age group,but everytime I execute the exe file it gives me the desired line and also the last line included everytime.Where did I go wrong with it including the last line everytime?
#include <iostream> using namespace std; int main() { double age; cout<<"Enter your age:"; cin>>age; if (0<age && age <6)
Opening large files in c++. In my application, i am trying to save video as long as users have space in harddisk. What I am trying to do is when user is recording video i am trying to append the video data in to the file. The problem is that every time file size reach over 2GB my software crashes.
How can I create in my C++ program so a user can paste a text and then choose a character on specific line and place?. The above thing isn't that important. I just want to place a character on a specific line and place.
I mean something like this:
Enter a character:
You choosed " / "
On which line do want the character?
You choosed "Line 1 and 2"
Where do you want the the to appear on the line? (left or right)
I have written a C++ program that is supposed to open two text files (prog2a.dat and prog2b.dat) and write the contents of a specified range of lines to an output file (outfile.dat). I wrote a program that based on the example we were given (to take lines 5-15 from the first file and lines 4-12 of the second file and merge them into the output file) works perfectly. However, after asking my professor for some clarification on another part of the assignment, I found out I have not done this correctly. I have the code written so that it will always output the range of lines I mentioned earlier, but the program is actually supposed to allow the user to merge the files from the command line using whatever range they want by typing in the following command:
prog2 in1 5-15 in2 4-12 outfile
But I'm not sure how to tweak my current program to allow this to be done.
Here is the code that I have written, keep in mind that this works properly for the way it is written, but not how it is supposed to work for the purposes of the command line (hopefully that makes sense):
#include <iostream> #include <fstream> #include <cstdlib> using namespace std; int main() { // Create output file std::ofstream outFile("outfile.dat", ios::out);
[Code] .....
Is there any simple way to make this work as I described using the command line? Also, I am supposed to break this up into three files, a header file, the program file, and a test file (the test file contains main() and should close the 3 open files and display any error messages), but I'm getting really confused as to what should go in the header file. I know the header file should contain class definitions and constructors, but don't really know how to make that work for this specific program?
The below program is supposed to display the contents of all files listed in the command line. When I try to run the program I get the fatal error "Debug Assertion Failed" Expression: file != NULL. I've done some researching on the matter and I gather it might be because I don't have any files listed in the command line?
How to enter files in the command line! I opened the Command Window in Windows XP and tried typing in "C> argc" and "% argc" (argc being the name of the file containing the below program) without any luck.
Code:
#include <stdio.h> #include <stdlib.h> int main (int argc, char * argv[]) { int ch; // int to hold EOF int count; FILE *fp; for(count = 1; count <= argc; count++) // agrc loop
I am having an issue with the statement "Both the input and output files' names should be read from the command line." I don't understand what this means or what I need to do.
I need to set a timer but still have the program processing info. I have looked into using sleep but when using this all processing stops to my knowledge. I tried to use the settimer method but was having some issues since I am not using a dialog based interface.
I've been coding for about 10 years now and it just got to my mind that I would like to learn about how I can create speech recognition software. I am mostly interested in doing this from ground 0 not because there are no other libs I can use, but it's more of an interest to me.
grasp this new and magnificent feature that's becoming more and more used nowadays.
My preference would be coding in C++ (more because of OOP that's why not C) and I don't really like Java.
I know what I'm getting involved in but I am thrilled and willing to go all the way to achieve my goal.
I have a problem with calculating the CPUtime. I am using CPLEX to solve linear programming with 3 diferent algorithms.
1-primal Simplex 2-Dual Simplex 3-Barrier Method.
When I am trying to use Barrier method, my clock does not give me a right processing time. I don't have any problem with the other two methods. The value I am getting from ticks2 is 0 when I am using Barrier solver unlike the two other methods that I am getting some small values (ticks1 is fine). Using time_t ticks1 , ticks2 and time(&ticks1) and finally difftime(ticks1,ticks2) also gives me an integer number which is not precise enough. I need a way to compute cputime with accuracy of at least 2 decimals.
clock_t ticks1,ticks2; double solvingtime if (solver.primopt==1){ ticks1=clock(); status=CPXprimopt(myEnv, myLP); ticks2=clock();
The error ive been getting with this code is after i go to the check out option, enter the data then click (ctrl + z). Then program glitches back and forth from the menu to the check out option continuously and im not entirely sure why.
Heres my code:
[#include <iostream> #include <windows.h> #include <iomanip> #include <fstream> using namespace std; void records(){ cout << setw(10) << "Ministry of Health Hospital System
