C 网页搜索中的分段错误

Question

所以我正在构建一个网络搜索引擎，我尝试运行它，但我遇到了 seg 错误。我不知道为什么。我知道它在我的一个结构中。我会试着解释一下。所以在 indexPage.c 中，有一个 typedefed 到 queryHelper 的结构，每次对 URL 进行索引时，它都会存储找到的所有单词以及该结构中的计数。所以现在我们转到 crawler.c，在 crawler.c 中，每次 URL 需要索引时，我们都会从 indexPage.c 调用 indexMyStuff。 indexMyStuff 的返回值就是前面讲的结构体。所以在爬虫中，我们需要通过爬取过程来存储和保存这些结构中的每一个。所以我创建了一个指向这些结构的指针，称为 structArray。现在，我的爬虫需要返回指向所有这些结构的指针以及指向每个被索引的 URL 的指针。为此，我创建了第二个类型定义为 queryHelperExt 的结构，它同时满足了这两个要求。现在，爬虫返回 queryHelperExt 并在 WebSearch.c 中将返回值传递给查询。这就是棘手的地方。因此，我创建了一个名为 search 的函数（这是我划分错误的地方），它在每个结构的所有单词中搜索用户要求的单词。如果它在整个 queryHelper 中发现一个单词出现，它将该结构保存到容器中。我在搜索功能中 if(word[c]==index[i]->words[x][z]) 在这一行出现了段错误。这需要考虑很多，但我希望有人知道出了什么问题。

   //-------------------------------indexPage.h----------------//
    #ifndef INDEX_H
    #define INDEX_H
    #include <stdlib.h>
    #include <stdio.h>
    #include <string.h>
    #include <ctype.h>

    struct wordControl{
        char** words;
        int* countArry;
    };
    typedef struct wordControl queryHelper;
    queryHelper *indexMyStuff(char* argv);
    #endif

   //-------------------------------indexPage.c----------------//
    #include "indexPage.h"

    #define ALPHA_SIZE      26
    #define MAX_BUFF_SIZE   300000
    #define MAX_WORD_SIZE   500

    /**
     * This is the main, self-reffering structure, that behaves as a
     * trie data structure. This program assuems that the programmer will conform to the
     * conventions of the structure described below. Each of the elements that are defined
     * in the trieIndex::children variable are required to be mapped with the correct
     * alphabetical order (see the variable's definition below for more information).
     *
     * @param count:int          - Amount of times the word count has indexed the current
     *                             node depth level.
     *
     * @param children:trieIndex - This is an array that will map the next child node
     *                             with the pointer to the next character with the letter
     *                             of the alphabet. Each letter in the array is stored in
     *                             the corresponding letter value. For example, every 'b'
     *                             character is stored in the children[1] location, since
     *                             it is 1 spot away from the 'a' character. In summery,
     *                             'a' is defined in the 0 location, and 'z' is defined
     *                             in the 25th location in the array.
     *
     * @see alphaToInt()
     * @see intToAlpha()
     */
    typedef struct trieIndex trieIndex_t;
    struct trieIndex {
        int count;
        trieIndex_t *children[ALPHA_SIZE];
    };

    /**
     * Method that indexes a specific URL and returns the resulting trieIndex_t variable that
     * contains all the words that were located in the document, as well as the amount of times
     * each word was located in the document.
     *
     * @param url:char - The desired url to be searched through and indexed
     * @see addWordOccurrence()
     */
    trieIndex_t *indexPage(const char* url);

    /**
     * Helper method, primarily used in indexPage, that fills a word inside of a desired
     * trieNode_t structure. If the word is already stored in the trie object, the count
     * is simply incremented.
     *
     * @param word:char - The desired word to insert into a trie object.
     * @param wordLength:int - The length of the desired word.
     * @param trieNode_t:trieNode_t - The desired node to insert the word into.
     * @see indexPage()
     */
    int addWordOccurrence(const char* word, const int wordLength, trieIndex_t *trieNode_t);

    /**
     * Function defined for recursion ruse that prints the the contents of an trieIndex_t node
     * in alphabetical order.
     *
     * @param trieNode_t:trieNode_t - The desired node to print the contents of.
     * @param buffer:char - The initial text, use this if you want to concatenate
     *                      a previous node result, otherwise, pass in an empty
     *                      char string ("").
     */
    queryHelper *printTrieContents(trieIndex_t *trieNode_t, char buffer[MAX_WORD_SIZE],queryHelper *structure);


    /**
     * Helper function defined for recursion, use this to remove any parent node and all
     * of its children from the memory. If you want to clear up an entire trieNode, then
     * pass in the upper-most parent.
     *
     * @param trieNode_t:trieNode_t - The desired node to clean up and all it's children.
     */
    void freeTrieMemory(trieIndex_t *trieNode_t);

    // Already-defined...
    int getText(const char* srcAddr, char* buffer, const int bufSize);

    /*=============================================================================
      |   
      | Custom Function/Function Prototypes
      |
     *===========================================================================*/

    /**
     * Function that creates a trieIndex_t node and returns the new value. The children
     * are automatically filled in and the count is set to 0 by default.
     *
     * @param trieIndex_t - The newly-created node.
     */
    trieIndex_t *createNode(void);

    /**
     * Inverse of intToAlpha()
     *
     * Helper function that returns a letter's corresponding integer value (the distance
     * from the 'a' character. Assumes that the character is lower-case.
     *
     * @param c:char - The desired character to convert.
     * @return int - The character-specified integer distance from 'a'
     */
    int alphaToInt(char c) {
        return (char)c - (char)'a';
    }

    /**
     * Inverse of alphaToInt()
     *
     * Helper funciton that returns the integer's corresponding character value (the distance
     * from the 'a' character. Assumes that the returning character is desired to be lower-case.
     *
     * @param i:int - The desired integer to convert.
     * @return char - The integer-specified character distance from 'a'
     */
    char intToAlpha(int i) {
        return  (char)('a' + i);
    }

    /*============================================================================= |
      | End of Custom Function/Function Prototypes
      |
     *===========================================================================*/

     queryHelper *indexMyStuff(char* argv) {



        queryHelper *myStruct = malloc(sizeof(struct wordControl)*1); 
        myStruct->words=malloc(sizeof(char*)*100);
        myStruct->countArry=malloc(sizeof(int)*100);

        trieIndex_t *myTrie = indexPage(argv);
        printTrieContents(myTrie, "",myStruct);
        freeTrieMemory(myTrie);
        return myStruct;

    }


    trieIndex_t *indexPage(const char* url) {

        // Print the URL we are about to index
        printf("%s\n", url);
        // Initialize a new trie object to store the goodness in
        trieIndex_t *returnTrie = createNode();

        // The first character buffer that stores the result of getText, the
        // second character buffer temporarly stores each word that is found
        // from the resultBuffer. We assume that the two buffers can be the
        // same size, just for the sake of brevity.
        char resultBuffer[MAX_BUFF_SIZE], temporaryBuffer[MAX_BUFF_SIZE];

        // Use the defiend function to read the page's text
        getText(url, resultBuffer,  MAX_BUFF_SIZE);

        // We now define two counter variables, as well as another
        // variable that keeps track of what the previous character
        // was in order to check when the contiguous word is over
        // in order to begin the new word.
        int i = 0, j = 0, prevWasChar = 0;

        // Loop through the character buffer until we reach the null
        // terminating character (gaurenteed, so we don't have to have
        // bounds on the index key).
        while( 1 ) {

            // Check to see if the character is alpha-numeric
            if ( isalpha(resultBuffer[i]) ) {

                // If this is the first character
                if ( !prevWasChar ) printf("\t");

                // Ensure the code knows that the character we are using is
                // actually numeric
                prevWasChar = 1;
                // Append to the temporary buffer
                temporaryBuffer[j] = tolower(resultBuffer[i]);
                // Finally, we will print the current character
                printf("%c", temporaryBuffer[j++]);

            } else if ( prevWasChar ) {

                // This code block will execute when the previous buffer has
                // encountered a non-numeric character after scanning through
                // many numeric characters. Essentially, this means that we
                // have a word that needs to be stored.
                // We start off by null-terminating the last character
                temporaryBuffer[j] = '\0';

                // Add the word to the main tie of words
                addWordOccurrence(temporaryBuffer, j, returnTrie);

                // Also, we must reset the values for the next word
                j = prevWasChar = 0;

                // Since the word is over, we will print a line break
                printf("\n");

                // Now move on to the next iteration
                continue;
            }

            // Increment the counter.
            if ( resultBuffer[i++] == '\0' ) break;
        }

        // Now we can return the newly-created trie parent that contains all the
        // index child nodes inside of it.
        return returnTrie;
    }

    int addWordOccurrence( const char* word, const int wordLength,
            trieIndex_t *pIndex ) {
        // Loop index
        int i;
        int charPos = 0;

        // Now we loop through all the characters in the current word,
        // Note: we could have just checked when the word reaches the
        // null-terminating string, but I wanted to stick with the variables
        // that were alread provided in the prototype
        for ( i = 0; i < wordLength; i++ ) {

            // We need to get the position of the character (the numerical
            // location of the character in the alphabet).
            charPos = alphaToInt(word[i]);

            // Check to see if the character is defined
            if ( pIndex->children[charPos] == NULL ) {
                // If not, then define it
                pIndex->children[charPos] = createNode();
                pIndex->children[charPos]->count = 0;
            }

            // Set the next character positon to the value of the current
            pIndex = pIndex->children[charPos];
        }

        // Increment the last node
        pIndex->count++;

        return 0;
    }

    queryHelper *printTrieContents(trieIndex_t *pNode, char oldBuffer[MAX_WORD_SIZE], queryHelper *structure) {
      // Alphabet counter
      int i = 0;
      // The new buffer that contains the concatenated string
      char newBuffer[MAX_WORD_SIZE];

      // Now, lets step through each of the characters. It is worthy to note that this
      // will work in alphabetical order, since the code starts counting from 0-25, and
      // each letter is stored in the character array as a mapping from a = 0 to z = 25,
      // this will automatically display them in aplhabetical order.
      for ( i = 0; i < ALPHA_SIZE; i++ ) {
        // Check if the ith character is not null
        if ( pNode->children[i] != NULL ) {

          // Now we concatenate the character at the end of the existing string
          snprintf(newBuffer, MAX_WORD_SIZE, "%s%c", oldBuffer, intToAlpha(i) );

          // If the current string is actually a word that was found in the indexing
          // process, we will print it here, as well as the amount of times that the
          // indexing function found the word.
          if ( pNode->children[i]->count > 0 ) {
        structure->words[i]=newBuffer;
        structure->countArry[i]=pNode->children[i]->count;

          }

          // Recurse through all the characters until we reach the leaves of every
          // one of the branches.
          printTrieContents( pNode->children[i], newBuffer, structure );

        }
      }
    return structure;
    }

    trieIndex_t *createNode(void) {

        // First, we allocate some memory for the new node
        trieIndex_t *pReturnNode = (trieIndex_t *)malloc(sizeof(trieIndex_t));

        // Check to make sure the memory has been properly allocated before working with
        // the new child node
        if ( pReturnNode != NULL ) {

            // Se the default letter to 0
            //pReturnNode->letter = 0;

            // Now we shall loop through the letters and set them all to NULL
            int i = 0;

            // Set all the children to NULL by default
            for ( i = 0; i < ALPHA_SIZE; i++ ) {
                pReturnNode->children[i] = NULL;
            }

            // Set theefault count value to 0
            pReturnNode->count = 0;
        }

        // Return the fresh child (yes, you read that right)
        return pReturnNode;
    }

    void freeTrieMemory(trieIndex_t *pNode) {

        // Alphabet counter
        int i;
        // Loop through all the possible child nodes
        for ( i = 0; i < ALPHA_SIZE; i++ ) {

            // Check to see if this child node has children
            if ( pNode->children[i] != NULL ) {
                // This parent contains one (maybe more) child node, so we can take advantage
                // of recursion to clear the node child, so we pass it to the same method again,
                // and the code will check to see if it has children, and do the same with its
                // potential children, until we encounter a leaf node.
                freeTrieMemory( pNode->children[i] );
            }
        }

        // The heart of this function
        free(pNode);
    }

    /* You should not need to modify this function */
    int getText(const char* srcAddr, char* buffer, const int bufSize){
        FILE *pipe;
        int bytesRead;

        snprintf(buffer, bufSize, "curl -s \"%s\" | python getText.py", srcAddr);

        pipe = popen(buffer, "r");
        if(pipe == NULL){
            fprintf(stderr, "ERROR: could not open the pipe for command %s\n",
                    buffer);
            return 0;
        }

        bytesRead = fread(buffer, sizeof(char), bufSize-1, pipe);
        buffer[bytesRead] = '\0';

        pclose(pipe);

        return bytesRead;
    }


   //-------------------------------Crawler.h------------------//
    #ifndef CRAWLER_H 
    #define CRAWLER_H
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include "indexPage.h"
    struct pointersWordControl{
    queryHelper **queryArray;
    char** URLs;
    };
    typedef struct pointersWordControl queryHelperExt;
    queryHelperExt  *crawler(const char* FILE_NAME, const int MAX_N);
    #endif  



   //-------------------------------Crawler.c------------------//
    #include "crawler.h"
    #include "indexPage.h"
    //My Carwler

    int checkIndex(char* url, char** urlArray, const int MAX_N);
    int getLink(const char* srcAddr, char* link, const int maxLinkLength);
    char** missionControl(const char* FILE_NAME, const int Max_N, queryHelper      **structArray);





    queryHelperExt *crawler(const char* FILE_NAME, const int Max_N)
    {

        queryHelper **structArray = malloc(sizeof(struct wordControl*)*50); 
        char** urlString;
        urlString= missionControl(FILE_NAME, Max_N, structArray);
        queryHelperExt *queryPass=malloc(sizeof(struct pointersWordControl)*1);
        queryPass->queryArray=structArray;
        queryPass->URLs=urlString; 
        return queryPass;

    }



    char** missionControl(const char* FILE_NAME, const int Max_N, queryHelper **structArray){ 

        const int  MAX_BUFFER = 1000;
        int cursor = 0;
        int x; 
        int count=0;
        char** urlArray = malloc(sizeof(char*)*Max_N);

        for(x=0; x<Max_N; x++) {
            urlArray[x] = NULL;
        }
        FILE *file = fopen(FILE_NAME, "r");
        char* url;
        url = malloc(sizeof(char)*1000);
        int* hoplimit;
        hoplimit = malloc(sizeof(int));
        char* reader;
        reader = malloc(sizeof(char)*1000);
        int numHops;
        while(fgets(reader,MAX_BUFFER ,file ) !=NULL && cursor<Max_N) {
            sscanf(reader, "%s %d", url, hoplimit);
            numHops=0;
            while (1) {
                if (checkIndex(url, urlArray, Max_N)) {
                    structArray[count]=indexMyStuff(url);
                    count++;
                    cursor++;
                }
                numHops++;
                if(numHops<=*hoplimit && cursor<Max_N) {

                    if (!getLink(url,url,MAX_BUFFER)) {
                        break;
                    }

                } else {
                    break;
                }
            }
        }

        free(url);
        free(hoplimit);
        free(reader);
        fclose(file);
        return urlArray;
    }

    int checkIndex(char* url, char** urlArray, const int MAX_N)
    {
        int i;
        for(i=0; i<MAX_N; i++) {
            if (urlArray[i]==NULL) {
                urlArray[i] = malloc(sizeof(char)*(strlen(url)+1));
                strcpy(urlArray[i], url);
                return 1;
            } else if (strcmp(urlArray[i],url)==0) {
                return 0;
            }
        }
        return 0;
    }


    int getLink(const char* srcAddr, char* link, const int maxLinkLength)
    {
        const int bufSize = 1000;
        char buffer[bufSize];

        int numLinks = 0;

        FILE *pipe;

        snprintf(buffer, bufSize, "curl -s \"%s\" | python getLinks.py", srcAddr);

        pipe = popen(buffer, "r");
        if(pipe == NULL) {
            fprintf(stderr, "ERROR: could not open the pipe for command %s\n",
                    buffer);
            return 0;
        }

        fscanf(pipe, "%d\n", &numLinks);

        if(numLinks > 0) {
            int linkNum;
            double r = (double)rand() / ((double)RAND_MAX + 1.0);

            for(linkNum=0; linkNum<numLinks; linkNum++) {
                fgets(buffer, bufSize, pipe);

                if(r < (linkNum + 1.0) / numLinks) {
                    break;
                }
            }

            /* copy the address from buffer to link */
            strncpy(link, buffer, maxLinkLength);
            link[maxLinkLength-1] = '\0';

            /* get rid of the newline */
            {
                char* pNewline = strchr(link, '\n');
                if(pNewline != NULL) {
                    *pNewline = '\0';
                }
            }
        }

        pclose(pipe);

        if(numLinks > 0) {
            return 1;
        } else {
            return 0;
        }
    }
   //---------------------------------query.h---------------------------------//
    #ifndef Query_H
    #define Query_H
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    void query();
    #endif

   //---------------------------------query.c---------------------------------//
    #include "query.h"
    #include "indexPage.h"
    #include "crawler.h"


    struct myStruct{
        queryHelper *helper;
        int index;
    };
    typedef struct myStruct find;


    void  query(queryHelperExt *myHelper){
        find **containers = malloc(sizeof(struct myStruct*)*100);
        char** urlsIndexed = myHelper->URLs;

        //This method will handle the count of all the urls indexed, but also the count of each 
        //word occurence of each word, then from there it will store each word found from the
        //indexed urls into an array.
    //  void organizeURLS(char** urls){
    //      int k;
    //      for(k=0;k<strlen(urls);k++){
    //          if(urls[k]!=NULL)
    //              urlCount++;
    //      }


    //  }


        void search(char *word, queryHelper **index){
            int i=0;int x=0; int c=0; int flag=0; int z=0; int y=0;
            while(index[i]!=NULL){
                while(index[i]->words[x]!=NULL){
                    for(;c<strlen(word);c++){
                        if(word[c]==index[i]->words[x][z]){
                            flag=1;
                        }else{flag=0;}
                        z++;
                    }
                    if (flag=1){
                        containers[y]->helper=index[i];
                        containers[y]->index=i;
                        y++;
                            }x++;
                }
                i++;
            }
        }















    void makeQuery(){

        queryHelper **indecies = myHelper->queryArray;
        char* inputStream = malloc(sizeof(char) * 100);
        const char delim[2] = " ";
        char *token;
        char* wordSearch= malloc(sizeof(char)*20);
        int res=0;
        int flag=0;
        int urlCount=0;
        do{

            printf("Enter a web queury:");
            fgets(inputStream, 100, stdin);
            int i;
            for(i=0;i<strlen(inputStream);i++){
                if (inputStream[i] =='\n')
                    inputStream[i]='\0';
            }
            if(strlen(inputStream)!=0){
                for(i=0;i<strlen(inputStream);i++){
                    if (inputStream[i]>'z' ||inputStream[i]<'a' &&  inputStream[i]!=' '){
                        printf("Please enter a query containing only lower-case letters.\n\n");
                        makeQuery();
                        break;

                    }
                }
                printf("Query is:\"%s\"\n", inputStream);
                printf("IDF scores are\n");
                token = strtok(inputStream,delim);

                while( token!= NULL){
                    strcpy(wordSearch, token);
                    search(wordSearch, indecies);
                    printf("IDF(%s): 0.0000000\n", token);
                    token = strtok(NULL,delim);
                }  printf("\n");

            }else {
                printf("Exiting the program\n");
                exit(0);
            }

        }while(inputStream!=" ");

    }
    makeQuery();
    }

    //---------------------------------webSearch.c---------------------------------//
      #include "crawler.h"
    #include "query.h"

    int main(int argc, char* argv[]){
    if (argc!=4){
    fprintf(stderr, "Not enough command arguments");
    return -1;
    }  
    char* file = argv[1];
    int numberHops =atoi( argv[2]);
    long seed =atoi( argv[3]);
    srand(seed);
    char** urls;
    query(crawler(file,numberHops));
    }

Answer 1

你在 search() 函数中有一个很常见的错误：

if (flag=1) {
  ...

将 1 分配给 flag，可能会覆盖它之前的 0 值，然后 if 在不应该执行的时候被执行。

你需要：

if (flag == 1) {
  ...

这很可能是导致崩溃的原因。

Answer 2

This code does not cleanly compile.
SO that is your first problem.

regarding the webcrawler.c file ...

//---------------------------------webSearch.c---------------------------------//
#include "crawler.h"
#include "query.h"

int main(int argc, char* argv[])
{
    if (argc!=4)
    {

        // <-- the following line ONLY states there is a problem with the 
        //     invocation of the executable.
        //     it actually needs to tell the user the correct invocation
        //     I.E. something that indicates the correct arguments
        fprintf(stderr, "Not enough command arguments");
        return -1;
    }

    char* file = argv[1];
    int numberHops =atoi( argv[2]);

    // <-- this seed is only if you want to have repeatability
    //     much better to #include <time.h> then srand(time(NULL));
    long seed =atoi( argv[3]);
    srand(seed);

    // <-- the following line will raise a compiler waring about unused variable
    char** urls;
    query(crawler(file,numberHops));

    // <-- the following, missing code, will cause the compiler to raise a warning
    //     about reaching the end of a function that is expected to return
    //     a value
}

Other things to note:

in the makequery function:

this line: char* inputStream = malloc(sizeof(char) * 100);
malloc can fail, so this line needs to be followed by
a check that 'inputStram' in not NULL

this line: char* wordSearch= malloc(sizeof(char)*20);
malloc can fail, so this line needs to be followed by
a check that 'wordSearch' is not NULL

this code block:
int i;
for(i=0;i<strlen(inputStream);i++)
{
            if (inputStream[i] =='\n')
            {
                inputStream[i]='\0';
            }
}

the only location that could/will contain a '\n' 
is at the very end of the inputStream char array
so all the above can be simplified to:
if( '\n' == inputStream[strlen(inputStream)] ) inputStream[strlen(inputStream)] = '\0';

this code block:
for(i=0;i<strlen(inputStream);i++)
{
      if (inputStream[i]>'z' ||inputStream[i]<'a' &&  inputStream[i]!=' ')
      {
            printf("Please enter a query containing only lower-case letters.\n\n");
            makeQuery();
            break;
      }
 }

 is implementing recursion, with no exit plan (cannot depend on the user's input)
 this recursion will result in a memory leak from the prior calls to malloc()

 this code block:
 else
 { // else, inputStream  only contained a '\n'
      printf("Exiting the program\n");
      exit(0);
 }
 will result in a memory leak due to not free'ing the 
 previously allocated memory from the two calls to malloc()

 this code block, past the end of the makequery function:
         makeQuery();
}
will result in the compiler raising a warming/error about unreachable code

this line: 
queryHelper **indecies = myHelper->queryArray;
refers to a typedef for struct wordControl
and the struct wordControl does not contain a field 'queryArray'
this will cause the compiler to raise an error message and fail the compile
perhaps what you were expecting is a reference to the struct pointerWordControl
which does have a field: queryHelper **queryArray
Also, there is no global instance of, not a passed in parameter named 'myHelper' variable/struct

there are LOTS more problems with the code, but this should get you started

Answer 3

regarding this code block in the search() function
                for(;c<strlen(word);c++)
                {
                    if(word[c]==index[i]->words[x][z])
                    { // then matching character found
                        flag=1;
                    }
                    else
                    {
                        flag=0;
                    }
                    z++;
                }
the result of this for code block is the flag is set depending on the LAST
char in the two strings (word) and index[i]->words[x][..]
probably not what you want to code to do.

regarding this code block:
                if (flag=1)
                {
                    containers[y]->helper=index[i];
                    containers[y]->index=i;
                    y++;
                }
the line( if( flag=1) does not compare flag to 1
rather, it sets the flag variable contents to 1
this is probably not what you want
and shows why the literal should ALWAYS be on the left.
Then the compiler would have caught this error
rather than someone having to spend a ton of time debugging 
your code to find it.
The code block needs to check 'y' against the size of the array 'containers'
to assure that the array is not overrun.
overrunning the array can/will result in a seg fault event


regarding this line:
 int i=0;int x=0; int c=0; int flag=0; int z=0; int y=0;
it takes a lot of time for a person to identify 
what each variable is being used for.
placing each variable in a separate line, with a trailing comment
would mean that no one would have to reverse engineer the code
to determine the purpose of each variable.
Note that  the variable 'z' is not reset between words,
so, after the first word, 'z' is indexing out into where ever
reading characters past the end of an array is undefined behaviour
and can easily result in a seg fault event