C++ :: Multiplethread - Sequentially Process Large Number Of Files
Jan 14, 2014
We have a program that sequentially processes a large number of files (currently about 700 expected to increase to about 1500). The program performs the same processing on each file (and doesn't involve any other file) which is io-bound and not cpu-bound. This process takes several hours and it is normally performed overnight.
I've refactored the program so that the processing for each file is done within its own thread (ie one thread created for the processing of one file). This gives rise to many hundreds of io-bound threads. This refactored program is working with no errors reported and has reduced the total processing time down to about 10 minutes.
Any problems that might arise having this number of threads (700 to 1500) created/running?
View 14 Replies
ADVERTISEMENT
Apr 14, 2014
Windows 7, 64 bits, Visual Studio 10.
I have a problem to read a large number of binary files, process them and store them under a new name. The program and routines go very well for 505 files. After reading 506 files, the program now refuses to read the next file. I have 16 Gb of memory and tried to close all other programs and restart the PC. it always stops after 506 files (512 files would be more understanding in a way...).
Here is my code. I have tried many things without success. This is only part of the loop that stops. The if test if (myfile.is_open() returns false by some reason. I can start the process again starting with the file that does not open and then it stops again after 506 files.
char * tfiBlock;
ifstream myfile (OrigFilename, ios::in|ios::binary|ios::ate);
if (myfile.is_open()) {
int lengde = myfile.tellg();
tfiBlock = new char [lengde];
//static char memblock [size];
[Code] .....
Clean up procedure:
delete[] tfiBlock;
Are there any limits to how many files that can be opened, or is it maybe someting to be set in the compiler?
View 5 Replies
View Related
May 21, 2014
I want to read a folder that contains large number of tif files. (say 1000 tif images) requirement is to read all files and store in a array/array pointer.
View 1 Replies
View Related
Apr 18, 2013
I have managed to make a program that permutates a string with repetition.
I ran it to permutate "abcdefghijklmnopqrstuvwxyz1234567890" with a limit of 5 characters.
This took a little over 5 hours for my pc to process this and I ended up with a .txt 403MB in size. Needless to say I am unable to open this .txt in notepad without Notepad.exe not responding and me having to end the process.
So what I want to do is modify my code to break up the output in to several files rather than one. Possibly all permutations starting with a in one file, b in another, etc.
Here is my current code:
#include <iostream>
#include <string>
#include <sstream>
[Code]....
As you can see it currently appends permutation.txt with all output. I would like it to make files like this permut_5char_a.txt, permut_5char_b.txt, etc.
View 12 Replies
View Related
Jul 5, 2012
I am reading in some large files to process. The files need to be parsed into multi-line sub units for processing (by a different physical process). My current setup is crude and parses the entire file into memory before beginning to process the sub units. This is fine as long as the file fits, so I get into trouble > 2GB or so when my machine flat runs out of memory. The simple thing to do is to only read in some of the file, process what was read in, and then read in more. I more or less know how to stop reading at some point in the file, but I'm not sure how to resume from that point later when I need more data. Is there a way to count lines and then getline() starting from a specific point in the file? From reading the getline() doc, it doesn't look like there is.
This is my current function,
Code:
void ParseFile(char *path, char type) {
// open input file
ifstream input_file(path);
if( !input_file.good() ) { cerr << "Failed to open " << path << endl; exit(ERRCODE_ERROR); }
int pos = 1;
[Code] ....
I need to remove the code to open the file from this function and open it elsewhere. I would like to call this every time I run out of data, if( tasks.empty() ), and get more data from the input file, but I don't know how to resume reading the input file from where I left off.
If this is not possible, I guess I would have to place a function call in while( input_file.good() ) to call out and process the data I have. When the current tasks list is finished, the list could be cleared and control returned into the while loop to continue reading input and start re-populating the list.
View 2 Replies
View Related
Mar 31, 2014
I am creating a simple log parser (loads a text file and filters out unnecessary information, but has the option to show the full log) and I'm running into an issue with fairly large log sizes (50+mgs). I have seen a few recommendations from a stream to memory manged files and even alternate 3rd party controls.
I foresee a few issues with any of the non-third party solutions (which I would prefer to avoid third-party add-ins) such as the scroll bar not correctly reporting the relative length or position of the complete text in the box (when displaying only a portion of the file at a time) and in the stream solution where you read on scroll (as necessary) have not only the same issues, but how do you resume reading in the middle of the file? This also all assumes I would be periodically clearing the RichTextBox to keep the memory usage down to avoid an OutOfMemoryException (which I have been running into.)
View 14 Replies
View Related
Feb 9, 2015
I'm trying to make sure my code is written in smaller modules, so my first step is to create my initialization process in and external file to load the necessary data from external sources and set up things like the content of drop down list boxes.
My first attempt failed to give me access to the combobox items add function so I moved that code back into the form1.h file:
Code:
public:
Form1(void) {
InitializeComponent();
//
//TODO: Add the constructor code here
//
}
void AddDate(char *date, int ID)
{
this->comboBox1->Items->Add("line 1");
}
It compiles fine, but the call to it in my Initialize.cpp file
Code:
MarketView::Form1::AddDate("abs",1);
Gives error C2352: 'MarketView::Form1::AddDate' : illegal call of non-static member function
OK, so I change "void AddDate" to "static void AddDate" and now get the error that "static member functions do not have 'this' pointers" so I go back to the "MarketView::Form1::comboBox1" situation where there is no legal syntax after "Box1 to get me to Items->Add
I've been an old fashion programmer for over 47 years. It seems as is the concept of programming computers has changed from the concepts of logic to memorization of complex syntax.
There has to be a simple answer to do this other than to write thousands of lines of code in one Form1.h file. I refuse to believe that the new programming concepts will not allow you to write code in smaller more manageable modules.
What is the proper syntax for breaking up the larger file into more manageable chucks?
View 12 Replies
View Related
May 12, 2013
I have a program that makes a directory (temporary) and in it some files that are necessary for computation while program is running. Now suppose a user decides to interrupt the program at some stage. what happens is that the program stops but my folder and files remain.
Q: How to override SIGINT command so that before the the execution stops the folder is deleted ?
View 4 Replies
View Related
Dec 22, 2012
I am making an application that processes multiple files (typically > 500) through a command prompt call. The way I start the command prompt app is by looping using a call to CreateProcess for each file that is to be processed. It works fine, except that I somehow loose 'connection' to my app so that
1: Windows says that the app. is 'Not Responding'
2: The Cprogress bar in my app is not updated before all files have been processed, even though there is a CreateProcess call and a Cprogress.StepIt() from the app for each file that needs processing.
I somehow suspect that the CPU gets swamped... I do not want that Windows starts to say that my app is 'not responding' and I want my Cprogress dialog bar to update according to the number of files that are progressed through.
I wonder if multithreading is the OK way to go instead of just kicking of series of CreateProcess calls? Maybe my CreateProcess is not ending correctly? It seems as if my app is 'not regaining control' before very late. The app never crashes though.
My CreateProcess code is listed below, maybe there can be a problem with it, or maybe I should do things in a different way? My app basicaly works as it never crashes, but with above mentioned problems it is NOT a pro solution...
void CMultiFilerDlg::ProcessFile(CString pdfFile) {
int i=0;
DWORD ProcID;
// Open file in text mode:
STARTUPINFO si;
PROCESS_INFORMATION pi;
char cmdArgs[2052];
[Code] .....
View 6 Replies
View Related
Jan 15, 2013
Opening large files in c++. In my application, i am trying to save video as long as users have space in harddisk. What I am trying to do is when user is recording video i am trying to append the video data in to the file. The problem is that every time file size reach over 2GB my software crashes.
View 5 Replies
View Related
Jan 22, 2015
storing int=9000000000; gives me another number. how can i store the value?my bad int number=9000000000;
View 3 Replies
View Related
Jan 20, 2013
I am trying to run a simulation with a large number of objects (mainly arrays and vectors). I am not sure where shall I define my objects: inside or outside of the main() function, like the following two structures:
(1) ---------------
//main.cpp
int main(){
array<double, 1000> a_1 = {};
array<double, 1000> a_2 = {};
......
func_1(a_1, a_2, ..., a_100);
return 0;
}
[Code]...
I know there is a question about scope. But besides this question (which seems have no difference between these two structures here), is there any difference in terms of execution performance or security issue?
View 3 Replies
View Related
Sep 15, 2013
I have a very large number in word ~6million digits long
I wish to make a program to check if it is prime.
View 5 Replies
View Related
Feb 16, 2014
Suppose i have a very large number stored as a string, say
std::string str = "1000000000000000000000000000000000001";
And i use std::stringstream to convert it to int like this
std::stringstream ss(str);
uint64_t i;
ss >> i;
Then I would be maxed out right. so how would one practically handle things like comparison of two such numbers.
I could think of 2 approaches :
1) I can compare the numbers character by character.
2) I can put the results of ss >> i; into an array then compare each element of array
would there be any other methods??
View 4 Replies
View Related
Sep 20, 2013
I want to have one case of my switch statement to print out how many times the user has chosen other cases in the switch statement. Such as "You have pressed 2 6 times and 4 3 times."
View 3 Replies
View Related
Nov 27, 2011
Code:
// declare child arrays
public string[,] childcolor;
public string[,] childgeneo;
// create dictionaries
public Dictionary<string, int> colors = new Dictionary<string, int>();
public string color; // name for dictionary
public Dictionary<string, string> geneo = new Dictionary<string, string>();
[Code]...
I'm getting "field' is used like a 'type' error on jw.childcolor which causes other parts to error out.
How do I fix this & why am I getting the error?
I want to convert the dictionaries to arrays to read & write sequentially.
View 4 Replies
View Related
Dec 8, 2014
I have a vector of int,
Code:
vector<int> row_numbers{1,2,3,4,5,6,7,8,9};
I want to sequentially remove one element at a time starting with the first. When the second element is removed, the first element needs to go back in. The sequence would look like
Code:
// original vector, row_numbers.size()=9
row_numbers{1,2,3,4,5,6,7,8,9};
// trimmed vector, row_numbers_trim.size()=8
[Code] .......
I have been working under the assumption that the best method would be to have row_numbers remain untouched and work on a copy. For each step in the sequence, you would create row_numbers_trim as a copy of row_numbers, and then remove an element from row_numbers_trim.
Code:
// position being removed
int counter = 0;
// copy original vector
row_numbers_trim = row_numbers;
// remove the first element from the copy
row_numbers_trim(row_numbers_trim.begin()+counter);
All you would have to do here is to increment counter in a loop. is there a better way?
View 8 Replies
View Related
Oct 10, 2014
Im having trouble on getting the quantity up of the variable "item.iqty". For example the current quantity is 5 and in this function, the user inputs a number and it should add to the variable "item.iqty". So if if the user inputs 2 then the current quantity should be 7 now but in my program it hasnt changed. its still 5
void addQty(FILE *fp)
{
int num, qty, r, c, n;
bool found;
[Code]....
View 3 Replies
View Related
Apr 5, 2013
I am writing a program to hide files behind other files using Alternate Data Streams in Windows NTFS file systems.
The program is as follows:
Code:
#include <stdio.h>
#include <stdlib.h>
int main(void){
char hostfile[75], hiddenfile[75], hiddenFileName[15] ;
printf("Enter the name(with extension) and path of the file whose behind you want to hide another file: ");
scanf("%75s", hostfile);
[Code]...
The complier is showing error as "Extra Perimeter in call to system" but I am not getting where?
View 4 Replies
View Related
Jan 16, 2014
I am writing a piece of code that requires me to display the last 1000 lines from a multiple text files (log files). FYI, I am running on Linux and using g++.
I have a log file from which - if it contains more than 1000 lines, I need to display the last 1000 lines. However, the log file could get rotated. So, in case where the current log file contains less than 1000 lines, I have to go to older log file and display the remaining. For e.g., if log got rotated and new log file contains 20 lines, I have to display the 980 lines from old log file + 20 from current log files.
What is the best way to do this? Even an outline algorithm will work.
View 6 Replies
View Related
Oct 7, 2014
How to get large fatorials in c++...in c++ i am getting the ouput upto maximum of 20!
View 4 Replies
View Related
Jul 15, 2013
In C++, the largest int value is 2147483647. So, an integer larger than this cannot be stored and processed as an integer. Similarly, if the sum or product of two positive integers is greater than 2147483647, the result will be incorrect. One way to store and manipulate large integers is to store each individual digit of the number in an array. Write a program that inputs two positive integers of, at most, 20 digits and outputs the sum of the numbers. If the sum of the numbers has more than 20 digits, output the sum with an appropriate message. Your program must, at least, contain a function to read and store a number into an array and another function to output the sum of the numbers. (Hint: Read numbers as strings and store the digits of the number in the reverse order.)
View 7 Replies
View Related
Apr 5, 2013
I have to make a function that can add two numbers of any size. So far I have:
#include <iostream>
#include <string>
struct number {
[Code]....
why I'm getting a segmentation fault? It's really dense because of all the conversions I had to do.
View 16 Replies
View Related
Aug 12, 2014
What I want to create is a program that sorts through a huge list (millions of lines).
I want it to get rid of any line that contains a word that isn't in an English dictionary.
Example list:
00sdfdsf
ahdadsg
angel
ksjflsjdf
green
green000
carrot
and it would go through millions like that, giving me only:
angel
green
carrot
as my new list.
How could I go about this? What extra programs would I need?
View 9 Replies
View Related
Apr 8, 2013
My programs gives a segmentation fault for large n (n=9999999). It works fine for small n. Where n = total numbers to sort
List<long>* Merge(List<long> *ParentList)
{
if(ParentList->length()==1)
{
[Code]....
View 4 Replies
View Related
Jul 29, 2014
I have designed a code and still i dont get the desired output ....
#include <iostream>
using namespace std;
int fact(int,int);
int arr[200]={1};
int main() {
int n,t,len=0,i=0,j,a;
cin>>t;
[Code] ....
View 3 Replies
View Related
