C++ :: Sort Huge Files (larger Than Available Memory)
May 30, 2012
I am trying to understand what techiques can be used to sort really huge files (larger than available memory). I did some googling and came across one technique.
1. Are there any better ways to get this done?
2. Is there some tweaking that can be done to make this itself better?
Large enough so that you get a lot of records, but small enough such that it will comfortably fit into memory
3. How do you decide on this value? Consider, memory is 4 GB and currently about 2GB is consumed, and file to sort is 10GB in size. (Consumed memory could of course change dynamically during execution - consumed more/less by other apps.)
View 5 Replies
ADVERTISEMENT
Mar 8, 2014
I’m writing an application for raw image processing but I cannot allocate the necessary block of memory, the following simple code gives me an allocation error.
double (*test)[4];
int block = 32747520;
test = new double[block][4];
off course with smaller block size (i.e. int block = 327475;) it works fine. Is there an allocation limit? How it is possible to deal with big blocks of memory?
View 2 Replies
View Related
Jun 14, 2012
i have a spec for fetch the files from server and predict the un-used files from the directory in this situation i am going to fetch the files from server it will return huge files, the problem is the cpu usage will increase while i am fetching large files, so i like to eliminate this scenario.
View 1 Replies
View Related
Jun 21, 2013
Why the size of a process grows larger in size for lots of small memory allocations. For example, say I have this struct, which is 16 bytes (for a 32 bit build):
Code:
struct Person {
int iID;
int iAge;
char * pForeName;
char * pSurName;
};
If I allocate memory like this:
Code:
LPBYTE lpB = new BYTE[sizeof(Person) * 1000000];
Then my process grows to 16,48KB in size, which is what I expected. However if I allocate memory like this:
Code:
Person * lpPerson;
for(int i = 0; i < 1000000; ++i)
lpPerson = new Person;
Then the process grows to 78,656KB, which I don't understand.
Additionally, I was surprised to find a vector acts more similarly to the first example. This code:
Code:
Person temp = { 0 };
std::vector<Person> people;
for(int i = 0; i < 1000000; ++i)
people.push_back(temp);
Only increases the process memory to 16,892.
View 5 Replies
View Related
Dec 11, 2014
I have written in a binary file using fstream an using sort value.
unsigned shortwidth=50,height=50;
file.write((char*)&width,sizeof(width));
file.write((char*)&height,sizeof(height));
When i try to read it back from fstream again there are some symbols (binary obviously). How can i get my values back? I want to read those symbols and in a way to convert them to my old width and height values.
View 6 Replies
View Related
Mar 26, 2013
Modify program 3 by saving the header file on your memory stick and remove it from the Header section of the VC++ in the Solutions window.
Code:
#include<stdio.h>
#define pi 3.141592654
#define A (num) * (pi) * (Radius) /* a macro defines a math equation */
float spherevolume (float);
main() {
[Code] ....
View 3 Replies
View Related
Jan 27, 2014
I have a data buffer project in Windows 7 x64 embedded OS (using Visual Studio 2008), that would work simply like that:
One writer application will send data (no network protocols, just procedure call on the same machine) to my app like 20 packages per second, each data packages will be approximately 3 MB size and comes with a timestamp tag.
My application will store each data item for 100 minutes and then remove. (So I can calculate the total size from beginnig no need for dynamic allocation etc...)
Meanwhile there will be up to 5 reader applications which will query data from my app via Timestamp tag and retreive data (no updates or deletitions on data by reader apps).
So since the data in my buffer app can grow over 50GB I don't think that shared memory is going to work for my case.
I'm thinking about using Boost Memory Mapped Files or Windows API Memory Mapped Files.
So theoratically I will crate a fixed size File on harddisk (lets say 50GB) and then map some portion to RAM write the data to this portion, if the reader applications wants to use the data which is mapped currently on memory, then they will use directly, otherwise they will map some other potion of the file to their address spaces and use from there...
My problem is I haven't use Map File concept at all and I'm not a C++ developer so I'm not very familiar with the language fundementals. I've searched tutorials of boost and msdn for windows as well but there is not too much code example.
So I don't know how to map some portion of the data to memory and manage it, also how to search data in file or memory according to the timestamp tag. Yes there are codes for creating files and mapping the whole file to memory but none for dividing it into regions, aligning or padding the data which I need for my case.
View 2 Replies
View Related
Feb 25, 2013
I can not understand huge pointer, how its working.
#include<stdio.h>/ *How its working &decleration*/
int main(){
int huge *a =(int huge *)0x59990005;
int huge *b =(int huge *)0x59980015;
if(a == b)
printf("power of pointer");
else
printf("power of c");
return 0;
}
View 4 Replies
View Related
Nov 3, 2013
I have been coding a while on a 2D random terrain game. How would I go about saving the maps? I have an array for blocks, lighting, background and background lighting. The lighting is done in real-time, so exclude that.
But with two 32000x3200 arrays, I still need to store 204800000 separate numbers in a file. Oh god. How can I have this? I could write down the separate numbers, but...yeah.
If it matters, this is made in freeglut.
View 2 Replies
View Related
Nov 22, 2013
Im trying to implement a way to use really big numbers to be able to add, substract and multiply them with the class I made. for example:
huge a = "45646464646455464654656"
huge b = "456464564646544646"
huge c = a+b;
[Code]....
View 13 Replies
View Related
Dec 31, 2012
I want to build a server which holds hundreds of thousands of active users in memory. To keep all the users organized i would like to store them in a Vector.
The problem is how i could quickly and easy find the object whenever i need it? All users will have a unique ID. Would it be possible to keep some form of a Vector Index on the unique id number?
View 2 Replies
View Related
Mar 12, 2014
How to write a simple function that will take 3 ints and find the sum of the higher 2? This is what i got so far:
int findsum(int a,int b,int c)// will find the highest int and return it to our main program {
int max,max2;// this sets our local variable max
// next we will find the larger of our first 2 variables
if( a>=b)
{ max=a;
[Code] ....
How to get the second highest number and add it to max
View 19 Replies
View Related
Oct 29, 2012
Here is the code:
int unsigned long a,b;
scanf("%lu",&a);
scanf("%lu",&b);
printf("%lu",a*b);
suppose we input a very large number on a and b (no more than 1000 digit)
like a = 1234567890123456789012345678901234567890
and b = 9876543210987654321098765432109876543210
it outputs 1, why is it?
It is probably the answer is too big, is there any way to show a*b?
View 1 Replies
View Related
Apr 11, 2013
On compilation the following program does not give any error and when i run this program then sometimes it gives the segmentation fault. Specially with the larger values. I made sure that the values i inputted are not crossing any range of data types i used. I cannot get what is making my code to give segmentation fault error.
Code:
#include <stdio.h>
#include <string.h>
#include <math.h>
#include <stdlib.h>
int main()
[Code] ....
View 4 Replies
View Related
Nov 25, 2014
#include <stdio.h>
#include <iostream>
#include <cstdlib>
[Code]....
View 1 Replies
View Related
Oct 12, 2012
We never got into any gui stuff. It was strictly console apps. That being said I have been able to create a few small apps. I would like to start to create GUI based apps but I am kind of overwhelmed at the choices. I dont know where to go from here. Since I have c++ experience I would think Visual C++ would be the choice but i have been reading a lot of forums and getting other info.
Is there another language I should look at the would allow me to convert my program quicker or easier?
The console program I already created is very very simple and want to make a gui for.
Quick break down.
Reads file
Changes file contents based on search string,
saves and closes file
and kills a windows service
exits
Only thing I want to add would be the ability to run the file from a remote site within the company LAN(not a priority though). Currently it needs to be run locally on the server.
View 5 Replies
View Related
Apr 3, 2013
here is the link to question: [URL] ....
below is my code:
friendship relations #48.cppfriendship relations #48.cpp
I have some question in regards to the b part 2. My code is working fine when the number of input is low. However when I input more larger input data to my program it start to hangs.
Here is the input: [URL] ....
Output: [URL] ....
View 1 Replies
View Related
Apr 19, 2013
I added this snippet of code to a larger project I use to read some numbers in from a file, but for some reason I'm getting a segmentation fault.
Code:
printf("starting main
");
FILE *cutoffs;
double cut1=0, cut2=0, cut3=0;
printf("trying to open file
The cutoffs.in file looks something like
Code: 3.475 3.875 4.025
I ran the gdb debugger, so I know it is occurring at that fscanf; however, it doesn't give any other details (to be honest I don't have much experience with debuggers :x) ..The values are being put into an array. Before I just had numbers initialized in the array by hand, but need to have 20 different runs of this code with different numbers each time...
View 2 Replies
View Related
Aug 19, 2013
Example radix sort function to sort an array of 64 bit unsigned integers. To allow for variable bin sizes, the array is scanned one time to create a matrix of 8 histograms of 256 counts each, corresponding to the number of instances of each possible 8 bit value in the 8 bytes of each integer, and the histograms are then converted into indices by summing the histograms counts. Then a radix sort is performed using the matrix of indices, post incrementing each index as it is used.
Code:
typedef unsigned long long UI64;
typedef unsigned long long *PUI64;
PUI64 RadixSort(PUI64 pData, PUI64 pTemp, size_t count) {
size_t mIndex[8][256] = {0};
/* index matrix */
PUI64 pDst, pSrc, pTmp;
size_t i,j,m,n;
UI64 u;
[Code]....
View 9 Replies
View Related
Jul 31, 2013
How can I concatenate two 2-dimensional int arrays into one larger 3-dimensional array. This question is also valid for the 3-dimensional vectors. I know the command for the one dimensional vector as:
std::vector<int> results;
results.reserve(arr1.size() + arr2.size());
results.insert(results.end(), arr1.begin(), arr1.end());
results.insert(results.end(), arr2.begin(), arr2.end());
and for the one dimensional array as:
int * result = new int[size1 + size2];
copy(arr1, arr1 + size1, result);
copy(arr2, arr2 + size2, result + size1);
But I do not know how to make a 3-dimensional array or vector.
View 3 Replies
View Related
Nov 14, 2014
Q. WAP to find the next palindrome number larger than the input number.
for eg:-
Input=25
Output=33
The program is giving correct output for number with all digits '9';
why is it not giving output.
#include<iostream>
#include<string>
using namespace std;
int main() {
int len,flag=1,count=0,num,ind;
[code]....
View 8 Replies
View Related
Mar 22, 2013
how to sort a 1D array using function sort().but if i have a 2D array how do i sort only 1 row of it (or column)?
View 2 Replies
View Related
Feb 19, 2014
You will write a program that uses a multidimensional array having 3 rows and 8 columns and sorts each of the rows using both a bubble sort and a selection sort.
You must declare the array inside of main. You will have a for loop containing the calls to bubbleSort and selectionSort. You need to pass into function bubbleSort and selectionSort the following: 1) each column of the multidimensional array, 2) the size of the column, and 3) a particular row number of the multidimensional array to be used for printing out the "pass" shown on the following pages.
I keep getting an error that the identifier for bubbleSort and selectionSort is not found. (Error C3861) Also, I feel like I'm missing something in int main() to get it to sort properly.
# include <iostream>
using namespace std;
int main()
{
[Code].....
View 14 Replies
View Related
Jul 11, 2013
I was wondering why, in C, the sizeof of a struct is larger than the the sum of all the sizeofs of it's members. It only seems to be by a few bytes, but as a bit of a perfectionist I fine this a bit annoying.
View 1 Replies
View Related
Apr 5, 2013
I am writing a program to hide files behind other files using Alternate Data Streams in Windows NTFS file systems.
The program is as follows:
Code:
#include <stdio.h>
#include <stdlib.h>
int main(void){
char hostfile[75], hiddenfile[75], hiddenFileName[15] ;
printf("Enter the name(with extension) and path of the file whose behind you want to hide another file: ");
scanf("%75s", hostfile);
[Code]...
The complier is showing error as "Extra Perimeter in call to system" but I am not getting where?
View 4 Replies
View Related
Jan 16, 2014
I am writing a piece of code that requires me to display the last 1000 lines from a multiple text files (log files). FYI, I am running on Linux and using g++.
I have a log file from which - if it contains more than 1000 lines, I need to display the last 1000 lines. However, the log file could get rotated. So, in case where the current log file contains less than 1000 lines, I have to go to older log file and display the remaining. For e.g., if log got rotated and new log file contains 20 lines, I have to display the 980 lines from old log file + 20 from current log files.
What is the best way to do this? Even an outline algorithm will work.
View 6 Replies
View Related
